Python multithreading and Global Interpreter Lock

Question

I recently came across the GIL present in Python according to which only a single thread can execute at a time and multithreading can not utilize all the cores.

Now in one of my projects, I used multithreading and lots of locks and semaphores So here my question is can I achieve the same thing if I don't use locks and semaphores? i.e if I remove the concurrency logics from my project.

Edit: What I want to know is, whether it's possible to attain same functionality if I remove concurrency logics, I know what is GIL and it prevents threads to use all the cores and only one thread is run at a time.

check this link :https://stackoverflow.com/questions/11277784/python-gil-and-multithreading — Amit G, Sep 14 '17 at 05:22
@AmitGupta It doesn't answer my question i.e Is there still a need to protecting the shared resources using locks and semaphores if all the threads are not running at the same time because of the presence of GIL. — VINAY CHAUHAN, Sep 14 '17 at 05:28
Possible duplicate of [What is a global interpreter lock (GIL)?](https://stackoverflow.com/questions/1294382/what-is-a-global-interpreter-lock-gil) — Anurag Misra, Sep 14 '17 at 05:30
@VINAYCHAUHAN Yes, if you have multiple threads and shared resources you must still serialise access to them using mutexes even in the presence of the Global Interpreter Lock — donkopotamus, Sep 14 '17 at 05:39
@donkopotamus Why is that ? will it not be having double locking mechanism which will further degrade the performance of the program. — VINAY CHAUHAN, Sep 14 '17 at 05:47
You never know when you will loose the GIL, it could be mid-way between an operation that consists of several byte-code instructions. — cdarke, Sep 14 '17 at 05:48

score 4 · Accepted Answer · answered Sep 14 '17 at 05:50

4

The Global Interpreter Lock ensures that only one thread is executing byte code at once. That execution could be interrupted at any time.

Consider this simple function which might be intended to atomically store related values to attributes on an instance x

def f(x, a, b):
    x.a, x.b = a, b

Here is its disassembly into bytecode

          0 LOAD_FAST                1 (a)
          3 LOAD_FAST                2 (b)
          6 ROT_TWO
          7 LOAD_FAST                0 (x)
         10 STORE_ATTR               0 (a)
         13 LOAD_FAST                0 (x)
         16 STORE_ATTR               1 (b)
         19 LOAD_CONST               0 (None)
         22 RETURN_VALUE

Suppose x is not protected by a mutex. Then any thread executing f(x, 1, 2) can easily be interrupted between storing a (at 10) and storing b (at 16). That interrupting thread will now see x in an inconsistent state.

answered Sep 14 '17 at 05:50

donkopotamus

22,114
2
48
60

So you are trying to say GIL doesn't switch the thread if it is holding a lock btw your answer does clear things for me. – VINAY CHAUHAN Sep 14 '17 at 05:59
@VINAYCHAUHAN No, I'm not saying that. The GIL will still attempt to switch thread ... but if you were using a mutex to protect `x`, which the original thread has `acquired`, then the new switched in thread cannot possibly `acquire` that mutex and see `x` in an inconsistent state. – donkopotamus Sep 14 '17 at 06:03
Well, I believe then it's a bigger problem, suppose I have 100s of threads and each accessing the same resource, now if there is a context switch while the thread holds a mutex then the whole program (all the threads) block till the context changes to the original thread and the lock is released. – VINAY CHAUHAN Sep 14 '17 at 06:10
@VINAYCHAUHAN Surprisingly, you might find the python developers have thought of this ... the GIL only applies to code interpreting bytecode. A mutex acquire is C-code that does not require the GIL. The new thread, upon attempting to acquire the mutex will block in C-code, relasing the GIL to another thread. You have not suddenly discovered a massive hole in python thread design that no one has noticed in the last decade. – donkopotamus Sep 14 '17 at 06:12
@donkopotamus I think to prevent the problem you mentioned in the example, we have GIL. Since in the case of GIL the complete assembly block will be part of locking and unlocking sequence.I think that GIL wouldn't allow any inconsistency, except in the case where you are killing the whole program altogether, which I guess no locking semantics in the world could prevent. – Amit G Sep 14 '17 at 06:19
@AmitGupta No, without a mutex the "complete assembly block" will **not** be part of the "locking and unlocking sequence". The GIL **will** allow inconsistency if there is no mutex being acquired and released around the changes to `x` – donkopotamus Sep 14 '17 at 06:32

Python multithreading and Global Interpreter Lock

1 Answers1