7

CPython uses a Global Interpreter Lock. Linux has removed all traces of the Big Kernel Lock. What is the alternative to these locks? How can a system make full use of a truly multi-core or multi-processor system without grinding everything to a halt?

rook
  • 66,304
  • 38
  • 162
  • 239
  • C code (invoked from CPython) can run "full-threaded" as long as it doesn't need to grab the GIL -- it's only a restriction on the CPython engine. I'm not sure if Jython has similar restrictions. –  Mar 03 '11 at 07:38
  • 2
    Dupe of several of the articles under the Related column. Like http://stackoverflow.com/questions/265687/why-the-global-interpreter-lock. – typo.pl Mar 03 '11 at 07:57
  • 1
    Jython and IronPython don't have the GIL. – Iacks Mar 03 '11 at 08:46
  • @lacks Because they rely on the JVM and the CLR to handle threads. – NullUserException Mar 03 '11 at 08:47
  • 1
    @Rook The first answer on typo's link explain what the alternative to a GIL could be. – NullUserException Mar 03 '11 at 08:49
  • 1
    I don't understand the comparison of CPython with Linux. That's apples and oranges. – David Heffernan Mar 03 '11 at 09:01
  • @David Heffernan yes perhaps an apples thread and and oranges thread. – rook Mar 03 '11 at 16:09
  • 1
    @Rook. Also read this: http://stackoverflow.com/questions/991904/why-is-there-no-gil-in-the-java-virtual-machine-why-does-python-need-one-so-bad/991917#991917 – NullUserException Mar 04 '11 at 01:40
  • @NullUserException that is interesting. This is why i asked this question. To better understand new an upcoming interpreter technologies. – rook Mar 04 '11 at 02:56

4 Answers4

4

A GIL wouldn't be necessary if python used a more advanced Garbage Collector like IBM's Recycler instated of a primitive reference counting method. This is something that Unladen Swallow is doing to improve the performance of python. A more prommising answer is Stackless Python, which uses its own micro-thread implementation instead of relying on the operating system like traditional CPython.

rook
  • 66,304
  • 38
  • 162
  • 239
  • ...except that Unladen Swallow is dead, so no. – Nicholas Riley Jun 10 '11 at 02:38
  • I added a comment, not an answer... the point is that someone could expect that CPython's GIL will go away sometime soon based on your answer. It won't. – Nicholas Riley Jun 10 '11 at 03:25
  • Would making python's reference counts atomic operations (via the appropriate assembly instructions) work also? (I assume it's not that simple or they would have done it already) – Jeremy Friesner Jun 10 '11 at 04:03
  • 1
    Right, it was attempted with Unladen Swallow, but that has been pretty much abandoned. It seemed to be too much work so the developers went and created another language instead ([Go](http://golang.org/)). – Keith Jun 10 '11 at 04:15
  • @Keith now that is interesting. – rook Jun 10 '11 at 07:39
  • [PyPy](http://pypy.org/) is the new faster Python being developed with help of the creator of Psyco. It's over 3 times as fast as Unladen was, but not near Cython or Shed Skin: http://geetduggal.wordpress.com/2010/11/25/speed-up-your-python-unladen-vs-shedskin-vs-pypy-vs-c/ – Cees Timmerman Mar 12 '12 at 11:38
2

The GIL is process specific, so you can get around it by launching several Python processes. The multiprocessing module provides an easy-to-use API for this.

Another way is to use C-extensions (or write your own) which release the GIL while doing the kind of data processing you need.

shang
  • 24,642
  • 3
  • 58
  • 86
1

Simple. Have no mutable state, much like Haskell and other functional programming languages do. Since nothing in memory needs to be changed, no global lock is ever needed.

Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • But GIL is for the implementation of interpreted languages, not the code in it, right? Haskell is usually compiled, and compilers can be in non-functional languages. Even if lists were banned and everyone had to use tuples in Python, it would still need garbage collection etc, wouldn't it? – Mark Apr 17 '17 at 17:11
0

You can get rid of the GIL in quite the same way the Linux guys got rid of the Big Kernel Lock: simply add lots of more fine grained locks or use atomic primitives that don't need locks.

The downside and main reason Python doesn't do it, is performance. For example the Tcl interpreter has no GIL but can be compiled threaded and non-threaded, if you use the threaded version, you get around 10-20% less performance than in the single threaded case. So unless you use threads, it is actually slower. There have been Python patches to add smaller locks, but those had even worse performance impacts, so were rejected.

It is just a trade-off, the python devs decided that single thread performance (and backward compatibility with C-extensions) is much more important than the option to use many threads on the python level. You can still use threads freely inside a C-extension, just not meaningfully on the python language level.

schlenk
  • 7,002
  • 1
  • 25
  • 29