working in python console while executing a boost::python module

Question

How can i keep using the console while executing a process from a boost::python module? I figured i have to use threading but I think I'm missing something.

import pk #my boost::python module from c++
import threading
t = threading.Thread(target=pk.showExample, args=())
t.start() 

This executes showExample, which runs a Window rendering 3D content. Now i would like to keep on coding in the python console while this window is running. The example above works to show the Window but fails to keep the console interactive. Any Ideas how to do it? Thanks for any suggestions.

Greetings Chris

Edit: I also tried to make Threads in the showExample() C++ code, didn't work as well. I probably have to make the console a thread, but I have not a clue how and can't find any helpful examples.

Edit2: to make the example more simple I implemented these c++ methods:

void Example::simpleWindow()
{
    int running = GL_TRUE;
    glfwInit();
    glfwOpenWindow(800,600, 8,8,8,8,24,8, GLFW_WINDOW);
    glewExperimental = GL_TRUE;
    glewInit();
    glEnable(GL_DEPTH_TEST);
    glEnable(GL_CULL_FACE);
    glCullFace(GL_BACK);
    while(running)
    {
        glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
        glfwSwapBuffers();
        running = !glfwGetKey(GLFW_KEY_ESC) && gkfwGetWindowParam(GLFW_OPENED);
    }
}

void Example::makeWindowThread()
{
    boost::thread t(simpleWindow);
    t.join();
}

There may be some useless lines of code (it was just copy paste of a part from the real method i want to use.) Both methods are static. If I start interactive console in a thread and start the pk.makeWindowThread() in python, i can't give input anymore. Doesn't work if I put the call of pk.makeWindowThread() in a python thread as well. (I am trying to print something in console while showing the window.

Answer 1

When trying to execute a process while keeping the console interactive, then consider using the subprocess or multiprocessing modules. When doing this within Boost.Python, it is probably more appropriate to execute a process in C++ using execv() family of functions.

When trying to spawn a thread while keeping the console interactive, then one must consider the Global Interpreter Lock (GIL). In short, the GIL is a mutex around the interpreter, preventing parallel operations to be performed on Python objects. Thus, at any point in time, a max of one thread, the one that has acquired the GIL, is allowed to perform operations on Python objects.

For multithreaded Python programs with no C or C++ threads, the CPython interpreter functions as a cooperative scheduler, enabling concurrency. Threads will yield control when Python knows the thread is about to perform a blocking call. For example, a thread will release the GIL within time.sleep(). Additionally, the interpreter will force a thread to yield control after certain criteria have been met. For example, after a thread has executed a certain amount of bytecode operations, the interpreter will force it to yield control, allowing other threads to execute.

C or C++ threads are sometimes referred to as alien threads in the Python documentation. The Python interpreter has no ability to force an alien thread to yield control by releasing the GIL. Therefore, alien threads are responsible for managing the GIL to permit concurrent or parallel execution with Python threads. With this in mind, lets examine some of the C++ code:

void Example::makeWindowThread()
{
  boost::thread t(simpleWindow);
  t.join();
}

This will spawn a thread, and thread::join() will block, waiting for the t thread to complete execution. If this function is exposed to Python via Boost.Python, then the calling thread will block. As only one Python thread is allowed to be executed at any point in time, the calling thread will own the GIL. Once the calling thread blocks on t.join(), all other Python threads will remain blocked, as the interpreter cannot force the thread to yield control. To enable other Python threads to run, the GIL should be released pre-join, and acquired post-join.

void Example::makeWindowThread()
{
  boost::thread t(simpleWindow);
  release GIL // allow other python threads to run.
  t.join();
  acquire GIL // execution is going to occur within the interpreter.
}

However, this will still cause the console to block waiting for the thread to complete execution. Instead, consider spawning the thread and detaching from it via thread::detach(). As the calling thread will no longer block, managing the GIL within Example::makeWindowThread is no longer necessary.

void Example::makeWindowThread()
{
  boost::thread(simpleWindow).detach();
}

For more details/examples of managing the GIL, please consider reading this answer for a basic implementation overview, and this answer for a much deeper dive into considerations one must take.

Answer 2

You have two options:

• start python with the -i flag, that will cause to drop it to the interactive interperter instead of exiting from the main thread

• start an interactive session manually:

  import code
  code.interact()
  

  The second option is particularily useful if you want to run the interactive session in it's own thread, as some libraries (like PyQt/PySide) don't like it when they arn't started from the main thread:

  from code import interact
  from threading import Thread
  Thread(target=interact, kwargs={'local': globals()}).start()
  ...  # start some mainloop which will block the main thread
  

  Passing local=globals() to interact is necessary so that you have access to the scope of the module, otherwise the interpreter session would only have access to the content of the thread's scope.