2

I used the multiprocessing framework to create several parallel sub-process (via JoinableQueue), but I just set up the logging (using the normal python logging module) in my main thread. And as I test the code, it seems that all the sub-processes are able to put their logs into the single logfile that I specified in the starting of my main process with no issues.

However, according to the python logging cookbook, it says that the module logging is only thread-safe, but not process-safe. It suggests to use:

  1. multiprocessing.logging (which does not has full functionality of logging);
  2. use mutliprocessing.Lock to serialize the wiring to logfile from sub-processes
  3. use logging.QueueHandler to send logs into a multiprocessing.Queue, and then have a dedicated logging thread in the main process to handling writing log records into logfile

All the suggested solutions make sense to me, and I actually was able to implement solution #3 - it worked, no issues.

But, I do have the question about what would be the issue if we do not handle this well. What bad consequence might happen if I did not do any of #1,2,3 (as I described in the first paragraph)? And how can I make those bad consequence happen (I'm curious to see them)?

Jay Zha
  • 123
  • 4

1 Answers1

1

Generally you want log writes to be atomic in some fashion. That is, in this context, when something writes a chunk of text to a log, that chunk appears together rather than being split up and intermixed with the content of other log entries. If multiple processes try to write to a file without some kind of mediation, it can result in such intermixing or even clobbering of the content.

To purposely cause such a thing, have several processes write to the log repeatedly and simultaneously without mediation (no locks or handling processes) just as the documentation suggests you shouldn't. The more processes and the longer (partially dependent on buffer sizes) the writes are, the more likely you'll get intermixing.

Ouroborus
  • 16,237
  • 4
  • 39
  • 62
  • Thanks (+1) @Ouroboros, so when you say intermixed, do you mean the following? Say sub-process A is writing ‘AAAAA’ into the logfile, while sub-process B is writing ‘BBBBB’; if there is control, we might get a logfile with content of ‘ABABBABBAA’? – Jay Zha Aug 10 '19 at 13:21
  • @YujieZha Yes, though, because of buffering, the chunks tend to be larger. – Ouroborus Aug 10 '19 at 22:37