2

I am having a similar issue to this person. I am unable to run a simple multiprocessing routine in the pathos module and receive a pickling error. Below is the code and error.

from pathos.multiprocessing import ProcessingPool
import dill
class ProcClass(object):
    def __init__(self):
        pass
    def f(self,x):
        return x*x
pc = ProcClass()
pl = ProcessingPool(3)
print pl.map(pc.f, range(10))

The returned error:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/multiprocessing/pool.py", line 320, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I have installed: pp, dill, pox and multiprocessing then installed pathos. The install works but always gives me this error:

WARNING: One of the following dependencies is unresolved: pp(ft) >=1.6.4.5 dill >=0.2.4 pox >=0.2.2 (multi)processing

Based on a response from the author of pathos to a similar question, it looks like there is a problem with the install. I have removed and reinstalled several times, each time verifying the proper dependancies are installed. I'm running on MacOS and using python 2.7. Any help will be greatly appreciated!

Community
  • 1
  • 1
RHam
  • 21
  • 1
  • 2
  • I'm the author of the above packages. Can you give more detail how you installed, and which versions you installed? I primarily use MacOS and python2.7, so it should be straightforward to test your install. – Mike McKerns Oct 08 '15 at 11:32
  • From your traceback, it looks like you are using the standard library versions of `multiprocessing`, `threading`, and `pickle`. Essentially, not seeing any of the packages you say you installed. – Mike McKerns Oct 08 '15 at 11:38
  • I first downloaded the source [pathos-master] (https://github.com/uqfoundation/pathos) and ran the python setup.py install and tried it but didn't work. Then I deleted multiprocessing (as well as pathos) from my site packages, installed the dependencies again for pathos and did a clean install of pathos. Still didn't work. – RHam Oct 08 '15 at 19:41
  • If I try to run with regular multiprocessing I do get a slightly different error: PicklingError: PicklingError: Can't pickle : attribute lookup __builtin__.instancemethod failed rather than the previous: Can't pickle : attribute lookup __builtin__.function failed. I don't know if this helps but there is a difference. – RHam Oct 08 '15 at 19:44
  • What I am asking is can you tell me how you installed all the dependencies and what are their versions? For example, this *should* work: `>$ pip install setuptools` then `>$ pip install git+https://github.com/uqfoundation/pathos.git@master`. Then you can check the versions by importing and looking at `pathos.__version__`, for example. You should also make sure you are installing to the same site-packages as the python you are using -- I know that's obvious, but I've seen folks do otherwise. – Mike McKerns Oct 09 '15 at 02:17
  • I verified that i am installing to the proper site-packages. I have the following versions installed there: pathos version: 0.2a1.dev0 dill version: 0.2.5.dev0 pp version: 1.6.4.5 pox version: 0.2.2 multiprocessing version: 0.70a1. In the pathos.multiprocessing.ProcessPool.map, self.serve returns a multiprocessing.pool.Pool object. I don't understand where dill is used instead of pickle. How/Where does this happen? – RHam Oct 09 '15 at 11:06

2 Answers2

0

It appears that you are missing a critical dependency. With pathos, you should use multiprocess, which is a fork of multiprocessing which uses the dill serializer.

>>> import multiprocessing
>>> multiprocessing.__version__
'0.70a1'
>>> import multiprocess   
>>> multiprocess.__version__
'0.70.4.dev0'
>>> 
>>> multiprocess.Pool().map(lambda x:x*x, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>
Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Thank you @Mike McKerns. This was a weird one. I tried to simply import multiprocess alone and could not because I received the following error from subprocess.py: ImportError: cannot import name _args_from_interpreter_flags. It turns out that I was unable to import _args_from_interpreter_flags from the subprocess module. This is because it was simply not there. I don't know why it was not. The solution was to download python2.7 and simply copy and paste subprocess.py. The sample code above works now. – RHam Oct 09 '15 at 17:22
0

I was having similar error. I changed my import part from:

from pathos.multiprocessing import ProcessingPool

to:

import pathos, multiprocess
from pathos.multiprocessing import ProcessingPool
import dill

and the code works. Not sure what's happening though. Maybe like Mike mentioned in another answer that pathos is finding the system bundled multiprocessing package instead of multiprocess. And maybe explicitly importing it somehow fixed the issue...

string_is_hard
  • 29
  • 2