Which operations can be done in parallel without grabbing the GIL?

Question

I'm looking at embedding Python in a multi-threaded C++ program and doing simple computations using numpy in parallel.

On other words, I'm using PyRun_SimpleString to call numpy functions. Is grabbing GIL required if I only write to existing numpy arrays, and take care not to modify the same array from different threads?

Edit, as mentioned in comments, this was addressed here: Global Interpreter Lock and access to data (eg. for NumPy arrays)

and a possible solution is to use numpy c interface directly using ctypes which takes care of releasing GIL: https://stackoverflow.com/a/5868051/419116

For posterity, here's what actually happens when you try to do "a*=2" without grabbing GIL:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000acc242 in PyImport_GetModuleDict ()
    at python_runtime/v2_7/Python/import.c:433
433     PyInterpreterState *interp = PyThreadState_GET()->interp;

#0  0x0000000000acc242 in PyImport_GetModuleDict ()
    at python_runtime/v2_7/Python/import.c:433
#1  0x0000000000accca1 in PyImport_AddModule (
    name=0x13e9255 <.L.str53> "__main__")
    at python_runtime/v2_7/Python/import.c:672
#2  0x0000000000aab29f in PyRun_SimpleStringFlags (
    command=0x13e8831 <.L.str10> "a*=2", flags=0x0)
    at python_runtime/v2_7/Python/pythonrun.c:976
#3  0x00000000008e3300 in main (argc=1, argv=0x7fffd2226ec0)
    at python_embed/interactive_nogil.cc:56

Here PyThreadState_GET is macro for _PyThreadState_Current which is a null-pointer. So apparently grabbing the GIL also involves setting up some Thread variables which are needed for any Python operation

@YaroslavBulatov any operation not involving Python objects... — Saullo G. P. Castro, Oct 18 '14 at 20:17

score 0 · Accepted Answer · answered Oct 19 '14 at 00:48

The GIL is required for PyRun_SimpleString because it invokes the Python interpreter. If you use PyRun_SimpleString without owning the GIL the process will segfault.

There are parts of the NumPy C API that do not require the GIL, but only the parts that do not touch Python objects or the interpreter. One example is the PyArray_DATA macro.

Which operations can be done in parallel without grabbing the GIL?

1 Answers1