Does using the subprocess module release the python GIL?

Question

When calling a linux binary which takes a relatively long time through Python's subprocess module, does this release the GIL?

I want to parallelise some code which calls a binary program from the command line. Is it better to use threads (through threading and a multiprocessing.pool.ThreadPool) or multiprocessing? My assumption is that if subprocess releases the GIL then choosing the threading option is better.

Please clarify. The current answers think you are worried about the child process itself somehow holding the GIL, but I think you are perhaps concerned that `subprocess.call()` or `subprocess.Popen(...).wait()` will block other threads in the caller. (They do not.) — pilcrow, Apr 29 '14 at 16:19
@pilcrow: As someone who come across this question from Google, I would suggest you turn your comment into an answer, because unlike the existing answers it addresses the actual question. — Rörd, May 11 '15 at 11:36

score 17 · Answer 1 · answered May 11 '15 at 22:10

When calling a linux binary which takes a relatively long time through Python's subprocess module, does this release the GIL?

Yes, it releases the Global Interpreter Lock (GIL) in the calling process.

As you are likely aware, on POSIX platforms subprocess offers convenience interfaces atop the "raw" components from fork, execve, and waitpid.

By inspection of the CPython 2.7.9 sources, fork and execve do not release the GIL. However, those calls do not block, so we'd not expect the GIL to be released.

waitpid of course does block, but we see it's implementation does give up the GIL using the ALLOW_THREADS macros:

static PyObject *
posix_waitpid(PyObject *self, PyObject *args)
{
....
Py_BEGIN_ALLOW_THREADS
pid = waitpid(pid, &status, options);
Py_END_ALLOW_THREADS
....

This could also be tested by calling out to some long running program like sleep from a demonstration multithreaded python script.

As a rule of thumb, CPython releases GIL while using blocking OS API such as `waitpid()`. There is nothing specific about methods from `subprocess` module. Note: `execve()` obviosly blocks (*in the child* after the fork in this case) -- it returns only on error. `fork()` is a special case: ["read this discussion to understand why you should avoid mixing multithreading and `fork()`](http://bugs.python.org/issue6721) (`fork()` immidiately followed by `exec()` is fine). — jfs, May 12 '15 at 18:05
@J.F.Sebastian: yes re: rule of thumb and the dangers of mixing threads and forks. I'd question the characterization of `execve()` as "blocking," however. A successful `execve` doesn't block the caller, it vaporizes the caller. — pilcrow, May 12 '15 at 19:37

score 7 · Answer 2 · edited May 23 '17 at 12:25

7

GIL doesn't span multiple processes. subprocess.Popen starts a new process. If it starts a Python process then it will have its own GIL.

You don't need multiple threads (or processes created by multiprocessing) if all you want is to run some linux binaries in parallel:

from subprocess import Popen

# start all processes
processes = [Popen(['program', str(i)]) for i in range(10)]
# now all processes run in parallel

# wait for processes to complete
for p in processes:
    p.wait()

You could use multiprocessing.ThreadPool to limit number of concurrently run programs.

edited May 23 '17 at 12:25

Community

1
1

answered Apr 29 '14 at 16:05

jfs

399,953
195
994
1,670

Should be `multiprocessing.Pool` – Danqi Wang Apr 30 '14 at 07:23
1

@DanqiWang: no. `multiprocessing` provides both a process-based Pool and a thread-based Pool with identical interfaces. Both can be used depending on circumstances. – jfs Apr 30 '14 at 11:42
I don't understand why you have a preference of `TheadPool` over `pool` here. `Pool` can also limit the number of concurrency and the running processes will not suffer from the GIL. Moreover, `ThreadPool` is poorly documented. – Danqi Wang May 02 '14 at 02:43
@DanqiWang: Popen starts *new* process; there are no GIL issues as the first paragraph in the answer says. You could use `from multiprocessing.dummy import Pool` (same thing as `ThreadPool`) then all you need to change your code from using threads to using processes is to remove `.dummy` from the import. The interface is identical. – jfs May 02 '14 at 04:54
1

Understood. Didn't notice that it's `Popen`. My bad. Thanks for explaining. – Danqi Wang May 02 '14 at 07:24
This won't work if you need to send data to the process or retrieve the output. For that you need to use communicat() which waits for the process to terminate. – T3rm1 Jun 28 '17 at 15:52
@T3rm1 wrong. 1- the code in the answer works as is. 2- Here's [how to adapt it, to get output from multiple processes concurrently.](https://stackoverflow.com/a/23616229) Though it has nothing to do with the GIL (which is released during blocking I/O operations) – jfs Jun 28 '17 at 16:10

score 1 · Answer 3 · edited May 23 '17 at 11:53

Since subprocess is for running executable (it is essentially a wrapper around os.fork() and os.execve()), it probably makes more sense to use it. You can use subprocess.Popen. Something like:

 import subprocess

 process = subprocess.Popen(["binary"])

This will run in as a separate process, hence not being affected by the GIL. You can then use the Popen.poll() method to check if child process has terminated:

if process.poll():
    # process has finished its work
    returncode = process.returncode

Just need to make sure you don't call any of the methods that wait for the process to finish its work (e.g. Popen.communicate()) to avoid your Python script blocking.

As mentioned in this answer

multiprocessing is for running functions within your existing (Python) code with support for more flexible communications among the family of processes. multiprocessing module is intended to provide interfaces and features which are very similar to threading while allowing CPython to scale your processing among multiple CPUs/cores despite the GIL.

So, given your use-case, subprocess seems to be the right choice.

`process.stdout.readlines()` may block forever if any of the child processes fill any of their stderr pipe buffers. If you want to read both stdout and stderr separately then you need [asynchronous approach: threads or non-blocking pipes or iocp on Windows](http://stackoverflow.com/q/375427/4279) — jfs, Apr 29 '14 at 16:11

Does using the subprocess module release the python GIL?

3 Answers3

Linked

Related