1

My understanding is that the typical GIL manipulations involve, e.g., blocking I/O operations. Hence one would want to release the lock before the I/O operation and reacquire it once it has completed.

I'm currently facing a different scenario with a C extension: I am creating X windows that are exposed to Python via the Canvas class. When the method show() is called on an instance, a new UI thread is started using PyThreads (with a call to PyThread_start_new_thread). This new thread is responsible for drawing on the X window, using the Python code specified in the on_draw method of a subclass of Canvas. A pure C event loop is started in the main thread that simply checks for events on the X window and, for the time being, only captures the WM_DELETE_EVENT.

So I have potentially many threads (one for each X window) that want to execute Python code and the main thread that does not execute any Python code at all.

How do I release/acquire the GIL in order to allow the UI threads to get into the interpreter orderly?

martineau
  • 119,623
  • 25
  • 170
  • 301
Phoenix87
  • 1,003
  • 11
  • 15
  • 1
    This article might be of interest: https://opensource.com/article/17/4/grok-gil – Jeremy Friesner May 19 '18 at 23:36
  • Thanks but I've already been through that post. I have the feeling that the key is in the PyEval_* functions. However, it seems that a call to PyEval_AcquireLock from a third thread causes `Fatal Python error: PyEval_AcquireLock: current thread state is NULL` and I cannot understand why. – Phoenix87 May 19 '18 at 23:50
  • Did you read this also? https://docs.python.org/2.4/api/threads.html – Jeremy Friesner May 20 '18 at 00:56
  • Yes, that too, and I couldn't figure out what exactly the thread state is and how, if possible at all, you could create one manually. – Phoenix87 May 20 '18 at 07:57

2 Answers2

1

The rule is easy: you need to hold the GIL to access Python machinery (any API starting with Py<...> and any PyObject).

So, you can release it whenever you don't need any of that.

Anything further than this is the fundamental problem of locking granularity: potential benefits vs locking overhead. There was an experiment for Py 1.4 to replace the GIL with more granular locks that failed exactly because the overhead proved prohibitive.

That's why it's typically released for code chunks involving call(s) to extental facilities that can take arbitrary time (especially if they involve waiting for external events) -- if you don't release the lock, Python will be just idling during this time.

Heeding this rule, you will get to your goal automatically: whenever a thread can't proceed further (whether it's I/O, signal from another thread, or even so much as a time.sleep() to avoid a busy loop), it will release the lock and allow other threads to proceed in its stead. The GIL assigning mechanism strives to be fair (see issue8299 for exploration on how fair it is), releasing the programmer from bothering about any bias stemming solely from the engine.

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
  • Thanks for the answer. I thought that the GIL was required to guard just Python bytecode, and that C APIs would be fine outside of it. Anyhow, I have tried what you suggest but, unless I've forgotten some lines of code, I am still getting a NULL current thread state in the third thread. This is where a SIGSEGV is generated: `0x569d26 mov rax,QWORD PTR [rip+0x4f71c3] # 0xa60ef0 <_PyThreadState_Current>`. After this instruction `rax` is 0. – Phoenix87 May 20 '18 at 07:59
  • @Phoenix87 you'll need to give a [mcve] for us to be able to tell anything specific. All I can say now is that `PyErr_Clear` is Python API so you need to hold the GIL; and that you need to get/release the GIL with `By_BEGIN/END_ALLOW_THREADS` macros that not only handles the lock but also saves/restores Python thread state from a local variable introduced by these macros. – ivan_pozdeev May 20 '18 at 13:10
  • @Phoenix87 Since this question asks about the general principle and your problem may have nothing to do with the GIL whatsoever (e.g. you might be using the API incorrectly), this probably warrants a separate question. – ivan_pozdeev May 20 '18 at 13:20
0

I think the problem stems from the fact that, in my opinion, the official documentation is a bit ambiguous on the meaning of Non-Python created threads. Quoting from it:

When threads are created using the dedicated Python APIs (such as the threading module), a thread state is automatically associated to them and the code showed above is therefore correct. However, when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.

I have highlighted in bold the parts that I find off-putting. As I have stated in the OP, I am calling PyThread_start_new_thread. Whilst this creates a new thread from C, this function is not part of a third-party library, but of the dedicated Python (C) APIs. Based on this assumption, I ruled out that I actually needed to use the PyGILState_Ensure/PyGILState_Release paradigm.

As far as I can tell from what I've seen with my experiments, a thread created from C with (just) PyThread_start_new_thread should be considered as a non-Python created thread.

Phoenix87
  • 1,003
  • 11
  • 15
  • AFAICS, `PyThread_start_new_thread` is not a part of public API, so if you use it, all bets are off. You should rather use fns from `threading` or `_thread` module. Or start a regular C thread, then convert it into Python thread as per https://stackoverflow.com/questions/29595222/multithreading-with-python-and-c-api . – ivan_pozdeev May 20 '18 at 20:00
  • How can one tell whether it is public or not if it can be used by importing the containing header file? – Phoenix87 May 20 '18 at 21:08
  • It's public if it's documented. If it's not documented, you just don't have any guarantess about its presence and behavior. I dunno why nonpublic symbols are in `Python.h`. Probably for simplicity 'cuz it's widely used around the codebase. You can ask at the `python-dev` mailing list about this. – ivan_pozdeev May 20 '18 at 22:10