0

I made several Windows shared libraries (dlls) to hide an embed python under the hood.

  • A.dll exports two C functions: start_python, stop_python which calls Py_Initialize and Py_Finalize respectively;
  • B.dll exports a B_func which calls some cython function declared by cdef public.
  • An application loads A.dll and B.dll by LoadLibrary and gets these exported functions, then the following codes cause segmentation fault:
A_start_python();  // in A.dll
B_func();          // in B.dll
A_stop_python();   // in A.dll

B_func will crash due to some NULL pointer access. It looks like the underlying cython function is called without a living python interpreter. Why?

The following works as expected:

B_start_python();  // in B.dll
B_func();          // in B.dll
B_stop_python();   // in B.dll


There are some other problems when I plays with the demo, for example,

{
    A_start_python();
    A_stop_python();

    B_start_python();
    B_func();        // segmentation fault
    B_stop_python();
}
{
    B_start_python();
    B_stop_python();

    B_start_python();
    B_func();         // AttributeError: 'NoneType' object has no attribute 'current_thread'
    B_stop_python();
}
{
    B_start_python();
    B_func();
    B_stop_python();
}

{
    B_start_python();
    B_func();         // AttributeError: 'NoneType' object has no attribute 'current_thread'
    B_stop_python();
}

It seems Python cannot be restarted. I found several other related questions, like this. But the behavior is not the same when starting/stopping python in the same dll or not: segmentation fault vs AttributeError.



This is A Simple yet Full Demo which reproduces these problems. An embed python included, no other dependencies required. Its a CLion project which is managed by a CMakeLists.txt.

This is just the Source Codes without any binaries


Some implementation details:

    bool start_python() {
        int status = PyImport_AppendInittab("py_funcs", PyInit_py_funcs);
        if (status == -1)
            return false;
        Py_Initialize();
        auto m = PyImport_ImportModule("py_funcs");
        return bool(m);
    }

    void stop_python() {
        Py_FinalizeEx();
    }

    void call_py_func() {
        std::cout << "To call_py_func. is Python initialized: " << Py_IsInitialized() << std::endl;
        py_func();   // cython cdef public function
    }
oz1
  • 938
  • 7
  • 18
  • Perhaps you link the two DLLs with different runtimes. Link both with Multi-threaded DLL (/MD) settings, or both with Debug Multi-threaded DLL (/MDd) settings. Other settings may or may not work. – n. m. could be an AI Nov 18 '22 at 07:38
  • Oh and there is absolutely no chance I will put your binaries on my computer. – n. m. could be an AI Nov 18 '22 at 07:41
  • Duplicate of https://stackoverflow.com/questions/70683041/add-python-modules-to-cython-for-use-in-c. importing the Cython module *is required* – DavidW Nov 18 '22 at 07:48
  • @DavidW Hi, DavidW. This is actually done in `start_python`, please see my edit. – oz1 Nov 18 '22 at 08:09
  • Hi, @n.m. I checked the link settings in the CMakeLists.txt and find no problems. I uploaded the source codes without any binaries, [link](https://drive.google.com/file/d/1BlIJefarbH8KOnXio6uGITPfg13jOO7A/view?usp=sharing) – oz1 Nov 18 '22 at 08:20
  • Not sure this is the whole issue but a lot of Python modules (including most Cython modules, and Numpy) don't cope with the interpreter being restarted and being reimported. Start the interpreter only once. – DavidW Nov 18 '22 at 09:24
  • @DavidW Thanks, I've known the interpreter restarting issue. But why can't `A_start_python(); B_func(); A_stop_python();` work? The interpreter starts once, and calls the python function, only in different dll. – oz1 Nov 18 '22 at 09:40

0 Answers0