6

I had some perfectly working python code which used multiprocessing module and loaded all 8 CPUs on my machine at 100%.

After I upgraded from Ubuntu 10.10 to 12.04 (the most evident thing, maybe I did something else that broke everything), it stopped working. After lots of debugging, I found that even in the simplest use case, both modules are only using 1 CPU:

from pylab import *
import multiprocessing as mp
from joblib import Parallel, delayed

def f(i):
    # Slow calculation
    x = 1
    for j in range(100000): x = cos(x)
    print i, x

if __name__ == '__main__':
    # Try to multiprocess with multiprocessing
    mp.Pool(processes=8).map(f, range(100))
    # Try to multiprocess with joblib
    Parallel(n_jobs=8)(delayed(f)(i) for i in range(100))

I need to use all 8 CPUs in my system. Any ideas of what I should look at to fix the issue?

starball
  • 20,030
  • 7
  • 43
  • 238
user1084871
  • 1,012
  • 1
  • 10
  • 12
  • could it be due to your Python version changing? I know that the default Python version changes from 2.6 -> 2.7 when going from Ubuntu 10 -> 12 – Cameron Sparr Mar 01 '13 at 23:26
  • 2
    You might be able to fix this problem using `taskset` - see my answer [here](http://stackoverflow.com/questions/15639779/python-what-determines-whether-different-processes-are-assigned-to-the-same-or?lq=1) – ali_m Mar 28 '13 at 23:51
  • 2
    Which value does `multiprocessing.cpu_count()` return? If it is not 8, there is a problem. – Roland Smith Mar 30 '13 at 19:39
  • FWIW, on my 12.10 running python 2.7.3, the `mp.Pool().map()` call consumes 100% of all 4 cpus in my system. – Robᵩ Apr 16 '13 at 14:16
  • I can't reproduce this on 13.04. – Apalala Apr 24 '13 at 18:15
  • 4
    If you have solved the problem then you really ought to write it as an answer and mark it as solved. This helps de-clutter the place and also helps the search engines. – Gareth Davidson Jan 04 '14 at 07:27
  • I tried the same procedure for a Jupyter notebook, and it does not work. In placed that `os` command as my first cell before I import any libraries. Any ideas on how to get this to work on Jupyter notebooks? – Jane Wayne Apr 23 '18 at 23:45
  • @user1084871 Please post your answer as an answer – endolith Mar 10 '21 at 04:27

1 Answers1

0

Solution moved from @user1084871's question post.

As @ali_m pointed out in a comment here and in the answer to Why does multiprocessing use only a single core after I import numpy? the problem is related to numpy messing up with cpu affinity. Calling

os.system('taskset -p 0xffffffff %d' % os.getpid())

before I do any multiprocessing solved the problem for me.

starball
  • 20,030
  • 7
  • 43
  • 238