Threading in Python using multiple cores

Question

As far as I know, Python's threading library uses POSIX threads for threading and it does not run on multiple cores. So is it possible that we implement a multicore threading system for Python threads using Open MP?

I think the `GIL` is the issue : https://wiki.python.org/moin/GlobalInterpreterLock — Assem, Apr 29 '15 at 15:29
See the wiki on [parallel processing](https://wiki.python.org/moin/ParallelProcessing) for ways to do it. — user3557327, Apr 29 '15 at 15:42
@Rayu please consider to mark an answer as accepted and upvote answers that helped you — Assem, Dec 20 '15 at 01:00

score 5 · Answer 1 · answered Apr 29 '15 at 15:45

CPython ("default" Python implementation) is not utilizing multiple cores because of Global Interpreter Lock. So every Python statement has to hold that lock.

But modules that are written in C may release interpreter lock before time-consuming operation. I.e. numpy does that: http://wiki.scipy.org/ParallelProgramming

They have handy example for that:

import numpy as np
import math

def f(x):
    print x
    # This statements hold GIL and cannot be run
    # in two parallel threads 
    y = [1]*10000000
    [math.exp(i) for i in y]

def g(x):
    print x
    # This statements fall to NumPy C code
    # than release GIL and can be multithreaded
    y = np.ones(10000000)
    np.exp(y)

Since OpenMP is also a tool for C, I think that is what you seek for.

score 3 · Answer 2 · answered Apr 29 '15 at 15:47

multiprocessing:

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

score 0 · Answer 3 · answered Apr 29 '15 at 15:45

0

CPython has a Global Interpreter Lock a.k.a GIL. The GIL ensures that only one thread runs in the interpreter at once. Can your multicore threading system beat the GIL?

More about GIL:

answered Apr 29 '15 at 15:45

Assem

11,574
5
59
97

Threading in Python using multiple cores

3 Answers3