2

I am trying to use the multiprocess module to parallelize my program.

Specifically, I am trying to achieve:

for i in range(something):
    p = multiprocess.Process(target=worker, args=(...))
    etc....

def worker(...):
    for i in range(something):
        make some system call and check for timeout

For the last line, I want to use threading, since subprocess can't support timeout checking, but I was wondering how GIL will affect the fact that they are running in processes spawned by multiprocess - i.e. will GIL kill the parallelization achieved through multiprocess by restricting each thread to run in one core.

rakrakrakrak
  • 69
  • 8
  • 2
    The GIL is a *per-process* artifact. Each running CPython process have its own GIL. It is only with same-process concurrency (eg. different threads) that there is GIL-lock concerns, and it works the same as it always does. – user2864740 Jun 22 '15 at 18:39
  • Thanks for the quick response - just to clarify, what you are saying is that since I already have multiple processes running, the threads I instantiate in each process will be running on a separate core (of course with the caveat that I only have so many cores). – rakrakrakrak Jun 22 '15 at 19:45
  • The threads within each process will act just like the threads within one process - and will be subject to all the same GIL restrictions. However, this does not imply the threads are on the same core. Nor is it guaranteed that the processes (and threads within) will be scheduled on different cores. The GIL is just a global lock when the Python code 'is actually being processed'. However, system-IO calls - and especially those with timeouts - *do not acquire the GIL* (they actually release it). It is only when the Python code resumes that the GIL is re-acquired by the engine. – user2864740 Jun 22 '15 at 20:33
  • Ref. http://stackoverflow.com/questions/23574367/understanding-python-gil-i-o-bound-vs-cpu-bound , http://stackoverflow.com/a/265731/2864740 , http://stackoverflow.com/a/991325/2864740 – user2864740 Jun 22 '15 at 20:37

0 Answers0