Can two concurrent Python processes run in Boost Python?

Question

I am trying to run concurrent Python scripts in a Python interpreter from boost::python. Here is a short sample program:

#include <Python.h>

#include <boost/python/exec.hpp>
#include <iostream>
#include <thread>

#include <boost/python/extract.hpp>
#include <boost/python/import.hpp>
#include <boost/python/object.hpp>

int main(int argc, char const *argv[]) {

  const char *prog = "def ack(m, n):\n"
                     "  if m == 0:\n"
                     "    return n + 1\n"
                     "  elif n == 0:\n"
                     "    return ack(m - 1, 1)\n"
                     "  else:\n"
                     "    return ack(m - 1, ack(m, n - 1))";

  Py_Initialize();
  try {
    std::thread t1([&prog]() {
      std::cout << "t1" << std::endl;

      boost::python::object mainModule = boost::python::import("__main__");
      boost::python::object mainNamespace = mainModule.attr("__dict__");

      boost::python::exec(prog, mainNamespace, mainNamespace);
      int val = boost::python::extract<int>(
          boost::python::eval("ack(3,3)", mainNamespace, mainNamespace));
      std::cout << "t1 result: " << val << std::endl;
    });

    std::thread t2([&prog]() {
      std::cout << "t2" << std::endl;

      boost::python::object mainModule = boost::python::import("__main__");
      boost::python::object mainNamespace = mainModule.attr("__dict__");

      boost::python::exec(prog, mainNamespace, mainNamespace);
      int val = boost::python::extract<int>(
          boost::python::eval("ack(3,4)", mainNamespace, mainNamespace));
      std::cout << "t2 result: " << val << std::endl;
    });

    t1.join();
    t2.join();
  } catch (boost::python::error_already_set const &e) {
    PyErr_Print();
  }

  return 0;
}

The problem is that the program fails intermittantly. I've tried it on two different Linux boxes. On one, it fails about 3/4 of the time; on the other about 1/10 of the time. The most common failure mesaage is the less than helpful:

RUN FINISHED; Segmentation fault; core dumped; real time: 60ms; user: 0ms; system: 0ms

There are several tempting calls in the Boost Pythnon documentation refering to concurrancy, but none of the combinations I've tried help. The global interpreter lock (GIL) seems to be desinged to allow C++ to access the same Python interpreter thread, unless I'm misunderstanding it. I want two completely independent Python interpreters running at the same time.

This example is close, but uses two separate C threads that occasionally call Python to do some work. I'm trying to run two separate Python processes concurrently.

This question is similar, although it gives less detail. In my case, the Python interpreters are used to tie together external processes that will take considerable time running on supercomputers; they do not need to call back into my C++ app. Once they complete, the results are collected and displayed by the C++ app from external dropping files.

The Python interpreter does not support concurrent execution of multiple threads. Multiple threads can run in the same interpreter, but only in a cooperative, time-sharing-style mode. Native threads can run concurrently, but their access to the interpreter and managed objects belonging to it must be serialized via the GIL. Boost cannot change that. — John Bollinger, Aug 03 '17 at 16:03
@JohnBollinger It seems like the only solution then is to do the concurrent Python work in separate processes rather than threads, as the example code above tries to do. Is that correct? — ukhat, Aug 03 '17 at 22:37
Yes, @ukhat, to the best of my knowledge, genuine concurrency of code running in the Python interpreter itself requires separate processes. — John Bollinger, Aug 04 '17 at 13:05

score 1 · Accepted Answer · answered Aug 11 '17 at 16:47

AFAIK, John is corrent. We never did find a way to run two concurrent Python projects in Boost Python. There are three ways to dodge this issue. Here's the first: Run two different Python interpreters.

#include <Python.h>

#include <boost/python/exec.hpp>
#include <iostream>
#include <thread>
#include <sys/wait.h>

#include <boost/python/extract.hpp>
#include <boost/python/import.hpp>
#include <boost/python/object.hpp>

void python (std::string fork, int m, int n) {
  const char *prog = "def ack(m, n):\n"
                     "  if m == 0:\n"
                     "    return n + 1\n"
                     "  elif n == 0:\n"
                     "    return ack(m - 1, 1)\n"
                     "  else:\n"
                     "    return ack(m - 1, ack(m, n - 1))";

  Py_Initialize();
  try {
    std::cout << fork << std::endl;

    boost::python::object mainModule = boost::python::import("__main__");
    boost::python::object mainNamespace = mainModule.attr("__dict__");

    std::stringstream commandstream;
    commandstream << "ack(" << m << "," << n << ")";
    std::string command = commandstream.str();
    boost::python::exec(prog, mainNamespace, mainNamespace);
    int val = boost::python::extract<int>(boost::python::eval(command.c_str(), mainNamespace, mainNamespace));
    std::cout << fork << " result: " << val << std::endl;
  } catch (boost::python::error_already_set const &e) {
    PyErr_Print();
  }
}

int main (int argc, char const *argv[]) {
  pid_t pid = fork();
  if (pid == 0) {
    python("f1", 3, 4);
  } else if (pid > 0) {
    python("f2", 3, 3);

    int status;
    waitpid(pid, &status, 0);
  } else {
    std::cout << "Fork failed." << std::endl;
  }

  return 0;
}

The second way, and the one we ended up using, is to place the code to run the Python interpreter in an external executable and run that.

The third is to block the thread until the first Python process finishes. That's viable if each Python process is expected to take a very short amount of time, but that was not the case in our application, so we rejected this alternative.

Can two concurrent Python processes run in Boost Python?

1 Answers1