simple python program with multiprocessing pathos library

Question

Followup with this question pathos multiprocessing cannot pickle

Running the code in one:

from pathos.multiprocessing import ProcessingPool as Pool

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(5)
    print(p.map(f, [1,2,3]))

Traceback (most recent call last):
  File "\\femto.niddk.nih.gov\C\All Projects\NMR High Pressure\Software\multiprocessing\multiprocessing_playground_2.py", line 12, in <module>
    results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']])
  File "C:\Python27\lib\site-packages\pathos\multiprocessing.py", line 137, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 567, in get
    raise self._value
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
>>>

Here is some more imformation:

>>> pathos.__version__
'0.2.2.dev0'
>>> import multiprocess
>>> multiprocess.__version__
'0.70.5'
>>> import multiprocessing
>>> multiprocessing.__version__
'0.70a1'
>>> multiprocess.Pool().map(lambda x:x*x, range(10))

Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    multiprocess.Pool().map(lambda x:x*x, range(10))
  File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 567, in get
    raise self._value
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

What is going on? Why isn't working? Isn't Pathos library supposed to use dill instead of pickling?

yes. >>> import dill>>> dill.__version__'0.2.7.1'. After searching online, I came to the conclusion that somehow pathos is still trying to use pickle instead. This was reported here https://github.com/uqfoundation/pathos/issues/121 . The solution there doesn't work for me. There is another question similar to mine: https://stackoverflow.com/questions/50747113/simple-python-program-with-multiprocessing-pathos-library?noredirect=1#comment88502552_50747113 — Valentyn, Jun 07 '18 at 18:14
I mistakenly put a wrong link address in the last line of previous comment: https://stackoverflow.com/questions/47250484/pathos-cpickle-error-on-python-2-7-13-14-using-windows-10?rq=1 — Valentyn, Jun 07 '18 at 18:21
it works if i use 'from pathos.pools import ParallelPool as Pool' instead of 'from pathos.multiprocessing import ProcessingPool as Pool' — Valentyn, Jun 07 '18 at 18:27

Valentyn · Accepted Answer · 2018-06-07T18:44:35.483

2

it works if I use

from pathos.pools import ParallelPool as Pool

instead of

from pathos.multiprocessing import ProcessingPool as Pool

of

from pathos.pools import ProcessPool as Pool

Who can explain what is the difference? All of these methods create different processes.

edited Jun 07 '18 at 18:44

answered Jun 07 '18 at 18:32

Valentyn

659
1
7
28

simple python program with multiprocessing pathos library

1 Answers1