4

When you create a Pool of processes using Python's multiprocessing, those processes will fork and globals in the parent process will show up in the child processes, noted in this question below:

How can I restrict the scope of a multiprocessing process?

This appears to include imported modules. That's a problem for modules that have side effects in __init__. Tensorflow is one such module, as soon as it's imported it tries to allocate memory to the GPU. This causes the sub process to crash because the parent already took that action.

Is there a good way to avoid the tensorflow module loading in the forked process?

Python 2.7 on Ubuntu (posix)

Community
  • 1
  • 1
David Parks
  • 30,789
  • 47
  • 185
  • 328
  • ubuntu forks, which means the child gets a copy-on-write view of the parent's memory space. It doesn't reimport modules. Windows is the opposite, it does reimport. – tdelaney Apr 26 '17 at 18:28
  • I've got a separate module that doesn't import tensorflow, I import the function I need to call from that module, then call `pool.apply_async(function_from_other_module)`. In the subprocess tensorflow is getting imported, I get a CUDA_OUT_OF_MEMORY error and see the sub process ID show up on the GPU (`nvidia-smi`). With one subprocess it actually allocates memory and doesn't crash which is how I see the process ID show up on the GPU. But I need multiple subprocesses and on the 2nd it does crash. I see tensorflow load once for each subprocess. – David Parks Apr 26 '17 at 18:42
  • Put the function you want to use in a module which doesn't import Tensorflow. – Peter Wood Apr 26 '17 at 22:51
  • Yup, first thing I tried, the module I'm importing my function from does not import tensorflow, only the parent does. – David Parks Apr 27 '17 at 00:06

2 Answers2

1

After much debugging I realize my problem statement wasn't really sufficient. The problem was that I did load tensorflow in one of the sub processes (I forgot!) and the sub processes needed to use the CPU only, not the GPU. I was forced to change environment variables to disable CUDA on the subprocesses with:

os.environ['CUDA_VISIBLE_DEVICES'] = ''
p = multiprocessing.Pool(processes=4)
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
David Parks
  • 30,789
  • 47
  • 185
  • 328
1

In addition to David Parks's self answer, it is safer to create Pool and worker instances before importing TensorFlow or modules that can be ruined by os.fork().

noname
  • 343
  • 4
  • 14