7

I am writing a C extension for Python, which should release the Global Interpreter Lock while it operates on data. I think I have understood the mechanism of the GIL fairly well, but one question remains: Can I access data in a Python object while the thread does not own the GIL? For example, I want to read data from a (big) NumPy array in the C function while I still want to allow other threads to do other things on the other CPU cores. The C function should

  • release the GIL with Py_BEGIN_ALLOW_THREADS
  • read and work on the data without using Python functions
  • even write data to previously constructed NumPy arrays
  • reacquire the GIL with Py_END_ALLOW_THREADS

Is this safe? Of course, other threads are not supposed to change the variables which the C function uses. But maybe there is one hidden source for errors: could the Python interpreter move an object, eg. by some sort of garbage collection, while the C function works on it in a separate thread?

To illustrate the question with a minimal example, consider the (minimal but complete) code below. Compile it (on Linux) with

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -fPIC -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/include/python2.7 -c gilexample.c -o gilexample.o
gcc -pthread -shared gilexample.o -o gilexample.so

and test it in Python with

import gilexample
gilexample.sum([1,2,3])

Is the code between Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS safe? It accesses the contents of a Python object, and I do not want to duplicate the (possibly large) array in memory.

#include <Python.h>
#include <numpy/arrayobject.h>

// The relevant function
static PyObject * sum(PyObject * const self, PyObject * const args) {
  PyObject * X;
  PyArg_ParseTuple(args, "O", &X);
  PyObject const * const X_double = PyArray_FROM_OTF(X, NPY_DOUBLE, NPY_ALIGNED);
  npy_intp const size = PyArray_SIZE(X_double);
  double * const data = (double *) PyArray_DATA(X_double);
  double sum = 0;

  Py_BEGIN_ALLOW_THREADS // IS THIS SAFE?

  npy_intp i;
  for (i=0; i<size; i++)
    sum += data[i];

  Py_END_ALLOW_THREADS

  Py_DECREF(X_double);
  return PyFloat_FromDouble(sum);
}

// Python interface code
// List the C methods that this extension provides.
static PyMethodDef gilexampleMethods[] = {
  {"sum", sum, METH_VARARGS},
  {NULL, NULL, 0, NULL}     /* Sentinel - marks the end of this structure */
};

// Tell Python about these methods.
PyMODINIT_FUNC initgilexample(void)  {
  (void) Py_InitModule("gilexample", gilexampleMethods);
  import_array();  // Must be present for NumPy.
}
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
Daniel
  • 924
  • 1
  • 11
  • 16
  • 3
    I did things like this in the past, and I found that the easiest way to make this work is using `ctypes` to call your C functions. Give your C functions a pure C interface without any reference to Python or NumPy, and write trivial wrappers in Python that accept NumPy arrays and translate them to the appropriate C parameters. I gave an example on how to do this in [this answer](http://stackoverflow.com/questions/5862915/passing-numpy-arrays-to-a-c-function-for-input-and-output/5868051#5868051). – Sven Marnach Jan 11 '12 at 21:19
  • @Sven: Do you know whether `ctypes` makes a working copy of the array in memory? (1) Yes. In this case, I don't want it since I am dealing with large input arrays. (2) No. Then my question whether you can lift the GIL remains valid. However, in case (2), the `ctypes` behavior would be a hint that lifting the GIL is probably not problematic, also in code which does not use ctypes. Does anyone know whether (1) or (2) holds? – Daniel Jan 11 '12 at 22:06
  • 7
    No, `ctypes` does not make a copy of the arrays. And it releases the GIL for you, so you don't have to care about it. The advantage of using `ctypes` is the simplicity -- you have to extract all necessary meta-information from the NumPy array while still in Python, and the GIL is released at just the right moment. I used this approach for concurrently accessing the data in NumPy arrays from multiple threads. (Note that concurrent write access to the same memory is never save.) – Sven Marnach Jan 11 '12 at 23:01
  • Thanks, Sven, this is the most helpful comment for me so far, even if it digresses a little from the original question to `ctypes`. If I could mark a comment as the ‘accepted answer’, this would be it. I'd like to know whether `ctypes` does more internally to protect the memory than what the explicit little example above does, but for a practical solution, it's sufficient for now to know that there exists a good way with `ctypes`. – Daniel Jan 12 '12 at 00:32
  • I wrote this as a comment rather than an answer because it digresses. `ctypes` doesn't do anything to protect the memory. The code for passing the NumPy array as a parameter to a C function is actually in NumPy, not in `ctypes` -- `ctypes` is part of the standard library and isn't aware of NumPy. All the code does is extract the pointer to the underlying data and pass it on to `ctypes`, so `ctypes` doesn't even know the size of the data. I'll write an more to-the-point answer shortly. – Sven Marnach Jan 12 '12 at 14:00

2 Answers2

6

Is this safe?

Strictly, no. I think you should move the calls to PyArray_SIZE and PyArray_DATA outside the GIL-less block; if you do that, you'll be operating on C data only. You might also want to increment the reference count on the object before going into the GIL-less block and decrement it afterwards.

After your edits, it should be safe. Don't forget to decrement the reference count afterwards.

Fred Foo
  • 355,277
  • 75
  • 744
  • 836
  • Thanks for the comment about `PyArray_SIZE` and `PyArray_DATA`. That was a mistake. I edited my question and moved the commands outside the block where the GIL is released. – Daniel Jan 11 '12 at 18:56
  • Now: is the code in its revised version safe? Can you elaborate why incrementing the reference count would change things? – Daniel Jan 11 '12 at 18:58
  • @Daniel: the reference count might be needed (though I'm not entirely sure in this case) because no other thread must deallocate the array. – Fred Foo Jan 11 '12 at 19:12
  • 2
    I checked the reference counts: `PyArray_FROM_OTF` increases the reference count already, so there is no need to do it manually. However, this brings to my attention that a `Py_DECREF` is in order at the end of my example code; otherwise there would be a memory leak. Thanks, larsmans! I corrected my original posting. So, as an answer to your original comment: if I incremented the reference counter manually, this would be unnecessary and would not change anything, since the reference counter is at least 1 and thus the object is safe from deallocation. – Daniel Jan 11 '12 at 19:33
4

Can I access data in a Python object while the thread does not own the GIL?

No you cannot.

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
  • Does that mean that I always have to copy the contents of a NumPy array, even if I only want to read the data in a thread? I hope that there is a way around it! Any suggestion? – Daniel Jan 11 '12 at 18:52