is boost::python not a thread-safe while running using OpenMP?

Question

I have tried some simple examples regarding implementing boost::python in OpenMP. Here is the one:

#include <omp.h>
#include <iostream>
#include <boost/python.hpp>

namespace p = boost::python;

int main() {
    Py_Initialize();
    p::object glob = p::import("glob");
    // glob.glob list more than 7000 files
    p::object paths = glob.attr("glob")("C:\\path\\to\\the\\dir\\location");

    #pragma omp parallel for default(shared) schedule(dynamic)
    for (int i = 0; i < p::len(paths); i++) {
        std::cout << p::extract<const char *>(p::str(paths[i])) << std::endl; 
    }
    std::cout << "finished: " << p::len(paths) << std::endl;
    return 0;
}

The code has worked if not using OpenMP. However, I have an error of 139 if using it. I also have tried using Python API (PyObject *), but it was still getting the same problem. I have pointed out that the problem is while we imported the Python modules.

Therefore, is there any possibility that we can use parallel programming on either boost::python or Python C API, especially while we use the Python modules on C++?

score 0 · Answer 1 · answered Sep 14 '21 at 08:40

I found the solution on this link: multithreading-with-python-and-c-api. The answer is we cannot truly create a multi-threading program using Python because of GIL.

I modified the code above into like this (I use the reference here based on the solution of the previous link):

#include <omp.h>
#include <iostream>
#include <boost/python.hpp>

namespace p = boost::python;

// -- I CITE THIS CODE SNIPPET BASED ON THE REFERENCE BELOW --
// link: https://gist.github.com/sterin/e8090d0451ab781a4e22
// initialize and clean up python
struct initialize
{
    initialize()
    {
        Py_InitializeEx(1);
        PyEval_InitThreads();
    }

    ~initialize()
    {
        Py_Finalize();
    }
};

// acquire GIL
class ensure_gil_state
{
public:
    ensure_gil_state()
    {
        _state = PyGILState_Ensure();
    }

    ~ensure_gil_state()
    {
        PyGILState_Release(_state);
    }

private:
    PyGILState_STATE _state;
};

// allow other threads to run
class enable_threads
{
public:
    enable_threads()
    {
        _state = PyEval_SaveThread();
    }

    ~enable_threads()
    {
        PyEval_RestoreThread(_state);
    }

private:
    PyThreadState* _state;
};
// --------------------------------------------------

int main() {
    initialize init;
    p::object glob = p::import("glob");
    // we have more than 7000 files
    p::object paths = glob.attr("glob")("C:\\path\\to\\the\\dir\\location");

    {
    enable_threads enable_threads_scope;
    #pragma omp parallel for
    for (int i = 0; i < p::len(paths); i++) {
        std::string pathi;
        {
            ensure_gil_state gil_scope;
            pathi = p::extract<const char *>(p::str(paths[i]));
            std::cout << i << " " << pathi << std::endl;
            std::cout << " -> out of scope" << std::endl;
        }
    }
    }
    std::cout << "finished: " << p::len(paths) << std::endl;
    return 0;
}

is boost::python not a thread-safe while running using OpenMP?

1 Answers1