{"repo":"jruere/multiprocessing-logging","free":true,"listed":false,"github":"https://github.com/jruere/multiprocessing-logging","clone":"git clone https://github.com/jruere/multiprocessing-logging.git","description":"Handler for logging from multiple processes","language":"Python","stars":395,"topics":["python","multiprocessing","logging","linux"],"license":"LGPL-3.0","category":"networking-infra","readme_excerpt":"multiprocessing-logging When using the multiprocessing module, logging becomes less useful since sub-processes should log to individual files/streams or there's the risk of records becoming garbled. This simple module implements a Handler that when set on the root Logger will handle tunneling the records to the main process so that they are handled correctly. It's currently tested in Linux on CPython 2.7, CPython 3.9+, and PyPy 2.7 and 3. Only works on POSIX systems and only Linux is supported. It does not work on Windows. Origin This library was taken verbatim from a StackOverflow post and extracted into a module so that I wouldn't have to copy the code in every project. Later, several improvements have been contributed. Usage Before you start logging but after you configure the logging framework (maybe with logging.basicConfig(...) ), do the following: and that's it. With multiprocessing.Pool When using a Pool, make sure install mp handler is called before the Pool is instantiated, for example: Problems The approach of this module relies on fork being used to create new processes. This start method is basically unsafe when also using threads, as this module does. The consequence is that there's a low probability of the application hanging when creating new processes. As a palliative, don't continuously create new processes. Instead, create a Pool once and reuse it.","default_branch":null,"files":null,"tree":[],"storefront":"/r/jruere","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jruere/multiprocessing-logging/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}