13

In the documentation for fit_generator() (docs: https://keras.io/models/sequential/#fit_generator) it says that the parameter use_multiprocessing accepts a bool that if set to True allows process-based threading.

It also says that the parameter workers is an integer that designates how many process to spin up if using process-based threading. Apparently it defaults to 1 (a single process based thread) and if set to 0 it will execute the generator on the main thread.

What I thought this meant was that if use_multiprocessing=True and workers > 0 (let's use 6 for an example) that it would spin up 6 processes running the generator independently. However, when I test this I think I must be misunderstanding something (see below).

My confusion arises from the fact that if I set use_multiprocessing to False and workers = 1 then in my task manager I can see that all 12 of my virtual cores are being utilized somewhat evenly and I am at about 50% CPU usage while training my model (for reference, I have an i7-8750H CPU with 6 cores that support virtualization and I have virtualization enabled in BIOS). If I increase the number of workers, the CPU usage goes to 100% and training is much faster. If I decrease the number of workers to 0 so that it runs on the main thread, I can see that all of my virtual cores are still being used, but it seems somewhat uneven and CPU usage is at about 36%.

Unfortunately, if I set multiprocessing = True, then I get a brokenpipe error. I have yet to fix this, but I'd like to better understand what I am trying to fix here.

If someone could please explain the difference between training with use_multiprocessing = True and use_multiprocessing = False, as well as when workers are = 0, 1, and >1 I would be very grateful. If it matters, I am using tensorflow (gpu version) as the backend for keras with python 3.6 in Spyder with the IPython Console.

My suspicion is that use_multiprocessing is actually enabling multiprocessing when True whereas workers>1 when use_multiprocessing=False is setting the number of threads, but that's just a guess.

chemdatafarmer
  • 333
  • 2
  • 3
  • 11
  • When `workers=0` multi cpu usage can be because some part of your preprocessing code can have parallel part, i.e. numpy functions. – mrgloom Aug 07 '19 at 09:27
  • Inside keras uses `multiprocessing.pool.ThreadPool` and `multiprocessing.Pool`. – mrgloom Aug 07 '19 at 09:29

1 Answers1

16

The only thing I know is that when use_multiprocessing=False and workers > 1, there are many parallel data loading threads (I'm not really good with these names, threads, processes, etc.). But there are five parallel fronts loading data to the queue (so, loading data is faster, but it doesn't affect the model's speed - this can be good when data loading takes too long).

Whenever I tried use_multiprocessing=True, everything got frozen.

Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • Thanks! So it sounds like when use_multiprocessing=False, workers are designating the number of threads to use. I'll have to read up on threading to better understand why this uses all my CPU cores regardless of the number of threads. Use_multiprocessing=True is still a mystery to me. Perhaps it is actually spinning up processes and changes the behavior of workers to set the number of processes to spin up? I'll ask a new question later on how to fix use_multiprocessing on windows after I double check it isn't already a solved question. – chemdatafarmer Feb 12 '19 at 00:32
  • @Daniel can you elaborate on `use_multiprocessing=True`? I've faced several issues same as you said frozen. How to use it properly? – Innat Mar 10 '21 at 03:52
  • 1
    And most of the time `WARNING:tensorflow:multiprocessing can interact badly with TensorFlow, causing nondeterministic deadlocks. For high performance data pipelines tf.data is recommended.` And many times freeze everything at epoch 1. – Innat Mar 10 '21 at 03:58
  • @M.Innat, I never used it. – Daniel Möller Mar 10 '21 at 21:28