1

I'm trying to build a multithreaded embedded python program (with cpython) and I'm facing difficulties with the global interpreter lock (GIL).

It seems that the GIL is released when PyObject_Call is called by the running thread. At this moment, another thread acquires the GIL and prevents the first one from finishing properly.

Here is the part of my code (lightened), I'm using Python 3.8 :

float pythonModule_process(PyObject *pProcessFunction, char* pFileBuffer, long lFileLen)
{
    float score = -1;
    PyGILState_STATE d_gstate = PyGILState_Ensure();
    /**** (STEP1) : ok here, our thread acquired the GIL ****/

    PyObject *pyBuf = NULL, *returnValue = NULL;
    pyBuf = Py_BuildValue("(y#L)", pFileBuffer, lFileLen, pthread_self() );    
    
    if (!pyBuf) {
        // (...)
        score= -1;
    } else {

    /**** PROBLEM IS HERE ! ****
     * Our thread is still holding the GIL but the PyObject_Call below won't get it. 
     * Instead, another thread will acquires the GIL (from step 1 above).
     * 
     ****/
        returnValue = PyObject_Call(pProcessFunction,pyBuf,NULL);;


        Py_DECREF(pyBuf); 
      
        if (!returnValue) {
            // (...)
            score = -1;
        } else {
            PyArg_Parse(returnValue, "f", &score);
        }    
  
      Py_DECREF(returnValue); 
    }
    
    PyGILState_Release(d_gstate);

    return score;
}

I believe this is because PyObject_Call creates a subprocess, which natively doesn't aquires the GIL ( more details here : Does using the subprocess module release the python GIL? )

So my question is : How could I ensure PyObject_Call does not release the GIL ? (apart from creating my own locking process) Or if I'm wrong, how should I process ?

Thank you for your help !

Julien

Julien
  • 2,747
  • 1
  • 12
  • 17
  • The GIL is not designed to work the way you're trying to use it. Why are you trying to maintain control of the GIL? – user2357112 Jan 15 '21 at 08:10
  • 1
    `PyObject_Call` does not create a subprocess. It performs the C-level equivalent of the Python function call operator. – user2357112 Jan 15 '21 at 08:12
  • 1
    If your code needs mutual exclusion for a critical section, you should use your own lock, not the GIL. The GIL is only intended to protect the Python language internals, and it may be released at points where such protection is not needed. – user2357112 Jan 15 '21 at 08:14

0 Answers0