6

I have a bit of python code that looks like this:

procs = cpu_count()-1
if serial or procs == 1:
    results = map(do_experiment, experiments)
else:
    pool = Pool(processes=procs)     
    results = pool.map(do_experiment, experiments)

It runs fine when I set the serial flag, but it gives the following error when the Pool is used. When I try to print something from do_experiment nothing shows up, so I can't try/catch there and print a stack trace. 

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 285, in _handle_tasks
    put(task)
TypeError: 'NoneType' object is not callable

What is a good way to proceed debugging this?

noio
  • 5,744
  • 7
  • 44
  • 61
  • This probably doesn't matter, but what is the return value from `cpu_count()` ? – mgilson May 30 '12 at 15:42
  • Depends on the system. 2 on my own laptop. 8 on the server. Either way; if `pool.map` is used in stead of `map`, things break. – noio May 31 '12 at 06:37

1 Answers1

17

I went back in my git history until I found a commit where things were still working.

I added a class to my code that extends dict so that keys can be accessed with a . (so dict.foo in stead of dict["foo"]. Multiprocessing did not take kindly to this, using an ordinary dict solved the problem.

noio
  • 5,744
  • 7
  • 44
  • 61
  • 1
    As a note, you really don't have to do it this way. Use a normal class and then do `self.__dict__.update()` – Voo May 31 '12 at 10:22
  • 1
    Wow, coincidentally me adding the exact same sort of class is what broke my multiprocessing too.. – Ryan Nov 21 '12 at 03:16
  • This sounds most unexpected. Any idea what the root cause of this bug might be? Is it a bug in the Python interpreter? – Craig McQueen Aug 07 '14 at 23:44
  • Perhaps it breaks the atomic access allowing deadlocks? – DevPlayer Nov 25 '16 at 02:56
  • 2
    Same error happens to me when I implemented some natural class attribute features that user can control, by defining ``__getattr__`` method, that way ended up with the ``NoneType` exception, while after I remove that method, error's gone. I guess the reason is the breaking of normal pickle way for class. – Tong Apr 05 '17 at 20:07