Fatal error during Py_Finalize in embedded Python application

Question

Thanks for your help with this -- variations on this question have been asked many times, but I haven't found a complete answer. I am adding embedded Python 3.4.2 to an existing simulator tool written in C++ using MS MFC classes. The application is multithreaded so that the user can execute Python scripts and interact with the simulator system.

How do I exit successfully? Am I using the GIL and thread state commands in the proper order? Am I ending the Python interpreter thread prematurely and breaking the Python thread consolidation mechanism?

My problem is that when I call Py_Finalize, it calls wait_for_thread_shutdown, then PyThreadState_Get, and hits a fatal error, "PyThreadState_Get: no current thread." Based on the point where the fatal error is detected, it seems to relate to the consolidation of threads at the end of a multithreaded embedded Python application.

I've condensed my code to clarify it and to eliminate anything that doesn't seem relevant. My apologies if I went too far or not far enough. The main thread initializes and finalizes Python.

BOOL CSimApp::InitInstance()
{
    ...
    // Initialize command table for appl. object and for documents
    int iReturn = PyImport_AppendInittab("sim", &PyInit_SimApp);
    iReturn = PyImport_AppendInittab("sim_doc", &PyInit_SimDoc);

    // Initialize Python and prepar to create threads
    _pHInstance = new CPyInstance();
    ...
}

int CSimApp::ExitInstance() 
{
    ...
    if (_pHInstance) {
        delete _pHInstance;
        _pHInstance = NULL;
    }
    ...
}

I'm using utility classes to create the Python instance (CPyInstance) and to manage the Python GIL (ACQUIRE_PY_GIL). When the application is initialized an instance of CPyInstance is also created. The class CPyInstance initializes and finalizes the Python thread management. Python Global lock management is accomplished with the ACQUIRE_PY_GIL and RELEASE_PY_GIL structures.

class CPyInstance
{
public:
    CPyInstance();
    ~CPyInstance();
    static PyThreadState * mainThreadState;
};

inline CPyInstance::CPyInstance()
{
    mainThreadState = NULL;
    Py_Initialize();
    PyEval_InitThreads();
    mainThreadState = PyThreadState_Get();
    PyEval_ReleaseLock();
}

inline CPyInstance::~CPyInstance()
{
    Py_Finalize();
}

static CPyInstance    *_pHInstance = NULL;

int PyExit()
{
    if (_pHInstance) {
        delete _pHInstance;
        _pHInstance = NULL;
    }
    return 0;
}

struct ACQUIRE_PY_GIL {
    PyGILState_STATE state;
    ACQUIRE_PY_GIL() { state = PyGILState_Ensure(); }
    ~ACQUIRE_PY_GIL() { PyGILState_Release(state); }
};

struct RELEASE_PY_GIL {
    PyThreadState *state;
    RELEASE_PY_GIL() {  state = PyEval_SaveThread(); }
    ~RELEASE_PY_GIL() { PyEval_RestoreThread(state); }
};

The Python interpreter thread is created in response to a Windows message handled by the CMainFrame window. The Python thread and interpreter runs in response to a user command. When the user finishes with the interpreter (Control-Z), the interpreter exits, the thread clears and deletes the Python thread state, and then the thread terminates itself.

void CMainFrame::OnOpenPythonInterpreter()
{
    // Create PyThread thread
    m_pPyThread = (CPyThread*)AfxBeginThread(RUNTIME_CLASS(CPyThread),
                    THREAD_PRIORITY_BELOW_NORMAL,0, CREATE_SUSPENDED);
    CMainFrame* mf = (CMainFrame*)theApp.m_pMainWnd;
    m_pPyThread->SetOwner(this,((CWnd*)mf)->GetSafeHwnd());
    m_pPyThread->CreateLocks(&m_PyThreadEvent,&m_PyThreadBusyMutex);
    m_pPyThread->ResumeThread();
}

The CPyThread class actually calls the Python interpreter. When the interpreter returns the GIL is released and the Python thread state is cleared and deleted. The thread terminates in response to the PostQuitMessage.

int CPyThread::Run() 
{
    PyEval_AcquireLock();
    PyInterpreterState * mainInterpreterState = CPyInstance::mainThreadState->interp;
    PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
    PyEval_ReleaseLock();

    try {
        ACQUIRE_PY_GIL    lock;
        FILE* fp1 = stdin;
        char *filename = "Embedded";
        PyRun_InteractiveLoop(fp1, filename);
    } catch(const std::exception &e) {
        safe_cout << "Exception in PyRun_InteractiveLoop: " << e.what() << "\n";
    } catch(...) {
        std::cout << "Exception in Python code: UNKNOWN\n";
    }

    PyThreadState_Clear(myThreadState);
    PyThreadState_Delete(myThreadState);

    ::PostQuitMessage(0);
    return 0;
}

int CPyThread::ExitInstance() 
{
    return CWinThread::ExitInstance();
}

At the suggestion of "user4815162342" I modified my ~CPyInstance() destructor to acquire the GIL before calling Py_Finalize(). Now my application appears to exit properly, thanks.

inline CPyInstance::~CPyInstance()
{
    try {
        PyGILState_STATE state = PyGILState_Ensure();
        Py_Finalize();
    } catch(const std::exception &e) {
        safe_cout << "Exception in ~CPyInstance(): " << e.what() << "\n";
    } catch(...) {
        std::cout << "Exception in Python code: UNKNOWN\n";
    }
}

Answer 1

You are calling Py_Finalize without holding the global interpreter lock. This is not allowed: the lock must be held for every Python API call, with the single exception of the call that acquires the GIL itself.

The ACQUIRE_PY_GIL RAII guard is not useful for this purpose, as it would try to release the GIL after Py_Finalize returns — in this case, you must call PyGILState_Ensure without the matching release.