0

I have a C++ application where we provide a python editor(which uses python 3.7), users can input and execute the python code in the editor. Following is the sample code where we are creating a new thread state and making it current before executing the script provided by users.

#include <python.h>

PyThreadState* globalthread;

void executescript()
{
    PyEval_SaveThread();
    PyThreadState* ts = PyThreadState_New(globalthread->interp);
    int gilstate = PyGILState_Check(); //check whether GIL is with current thread or not
    if (!gilstate)
    {
        PyEval_RestoreThread(ts);
    }
    else
    {
        
        PyThreadState_Swap(ts);
    }
    // get the thread state with which GIL is currently assigned
    PyThreadState * gilthreadstate = PyGILState_GetThisThreadState(); 
    //----> the above line return pointer to global thread state (gilthreadstate == globalthread)
    //----> which means the GIL is not acquired by the new current thread
    gilstate = PyGILState_Check(); //results in zero as GIL is not with current thread
    std::string str = "def script():\n\timport sys\n\tsys.path.append('C:\\Python\\Python37\\Lib\\site-packages')\n\tprint(sys.path)\n\timport numpy\n\tarr = numpy.array([1, 2, 3, 4, 5])\n\tprint(arr)\nscript()";
    PyRun_SimpleString(str.c_str());
    PyThreadState_Clear(ts);
    PyThreadState_DeleteCurrent();
    PyEval_RestoreThread(globalthread);
    //following stmt returns 1 as the current thread is global thread and it has GIL with it
    gilstate = PyGILState_Check(); 
}

int main()
{
    Py_Initialize();
    PyEval_InitThreads();
    globalthread = PyThreadState_Get();
    executescript();
    PyThreadState_Swap(globalthread);
    Py_FinalizeEx();
    return 0;
}

The new current thread is not acquiring the GIL when PyEval_RestoreThread or PyThreadState_Swap is called on it. Python scripts having import numpy or import pandas will end in deadlock due to this. Can someone let me know how to acquire GIL for the new thread state or where the code is wrong?

Revanth
  • 59
  • 8
  • How does `PyRun_SimpleString()` run Python? What is the callstack of the hang? – Thomas Weller Nov 09 '21 at 09:19
  • @ThomasWeller `PyRun_SimpleString()` takes a string that has script entered by the user through our editor. Before calling this we initialize the python framework using `Py_Initialize` . As requested the call stack is attached to the question. – Revanth Nov 09 '21 at 10:33
  • 1
    Are you calling `Py_Initialize` multiple times? If so I'm pretty sure this question is yet another duplicate of https://stackoverflow.com/questions/7676314/py-initialize-py-finalize-not-working-twice-with-numpy – DavidW Nov 09 '21 at 10:46
  • @DavidW `Py_Initialize` is called only once. – Revanth Nov 09 '21 at 11:43
  • @DavidW, looks like the issue is with the usage of python threads. I will change my question accordingly. – Revanth Nov 10 '21 at 07:05
  • None of that looks right. `PyEval_ReleaseLock` is deprecated since Python 3.2. `PyEval_SaveThread` and `PyEval_RestoreThread` are usually used as a pair to surround a "no GIL" block but you're doing something else entirely. I just don't understand what you're trying to do here. – DavidW Nov 10 '21 at 07:27
  • @DavidW this is the existing code in our application. I'm not sure why it's been done like this(creating a new thread) and I'm new to using embed python. I see after creating a new thread using `PyThreadState_New` and making it as the current thread return `PyGILState_Check` as zero, I feel this is causing the deadlock. I'm not sure how to pass the GIL to a new thread. – Revanth Nov 10 '21 at 08:08
  • What is the program supposed to do? Run a script in a separate thread? – n. m. could be an AI Nov 10 '21 at 10:20
  • @n.1.8e9-where's-my-sharem. Yes – Revanth Nov 10 '21 at 11:20
  • I have edited my question with meaningful description and code – Revanth Nov 10 '21 at 11:30
  • 1
    Perhaps look at [this](https://stackoverflow.com/questions/29595222/multithreading-with-python-and-c-api). – n. m. could be an AI Nov 10 '21 at 11:31
  • Creating a separate C++ thread and instantiating PyThreadState in the thread resolved the issue. Thank you @n.1.8e9-where's-my-sharem for your help. – Revanth Nov 10 '21 at 12:49

1 Answers1

1

Comment by hn-1-8e9-wheres-my-share-m has resolved the issue by creating a new C++ thread. Here is my modified code.

#include <python.h>
#include <thread>
PyThreadState* globalthread;

void execute()
{
    PyThreadState* ts = PyThreadState_New(globalthread->interp);

    int gilstate = PyGILState_Check();
    if (!gilstate)
    {
        PyEval_RestoreThread(ts);
    }
    else
    {

        PyThreadState_Swap(ts);
    }
    std::string str = "def script():\n\timport sys\n\tsys.path.append('C:\\Python\\Python37\\Lib\\site-packages')\n\tprint(sys.path)\n\timport numpy\n\tarr = numpy.array([1, 2, 3, 4, 5])\n\tprint(arr)\nscript()";
    PyRun_SimpleString(str.c_str());
    PyThreadState_Clear(ts);
    PyThreadState_DeleteCurrent();
}

int main()
{
    Py_Initialize();
    globalthread = PyThreadState_Get();
    PyEval_SaveThread();
    std::thread t(executePythonScript);
    t.join();
    PyEval_RestoreThread(globalthread);
    //PyThreadState_Swap(globalthread);
    Py_FinalizeEx();
    return 0;
}

Thank you for pointing out the post.

Revanth
  • 59
  • 8