12

So I'm trying to speed up my computation time by doing a little bit multiprocessing

I'm trying to use the pool workers.

At the top of my code I have 

import Singal as s
import multiprocessing as mp
def wrapper(Channel):
    Noise_Frequincies = []
    for i in range(1,125):
        Noise_Frequincies.append(60.0*float(i))
    Noise_Frequincies.append(180.0)
    filter1 = s.Noise_Reduction(Sample_Rate,Noise_Frequincies,Channel)
    return filter1

Then when the time comes I use

Both_Channels = [Chan1, Chan2]
results = mp.Pool(2).map(wrapper,Both_Channels)
filter1 = results[0]
filter2 = results[1]

I get the following error 

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run
self.__target(*self.__args, **self.__kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Chan1 and Chan2 are arrays of my signal, and I'm trying to filter out some noise out of each. I'm new to multiprocessing so I apologize if this is a dumb error

Cate Daniel
  • 724
  • 2
  • 14
  • 30
  • possible duplicate of [Can't pickle when using python's multiprocessing Pool.map()](http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma) – zmo Mar 03 '14 at 21:40
  • 1
    I don't use custom classes, and that's where his issue is (at least as far as I can tell) – Cate Daniel Mar 03 '14 at 21:42
  • 1
    @user2352742 The issue is that not all objects are picklable. Which objects aren't picklable? Just read [the docs](http://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled). – Bakuriu Mar 03 '14 at 22:16
  • can you tell more about how are defined Chan1/Chan2? what are they? – zmo Mar 03 '14 at 22:53
  • Chan1 and Chan2 is simply an array of numbers. They come from a series of binary data files, then trimmed to be a power of 2. – Cate Daniel Mar 03 '14 at 23:09

2 Answers2

5

I marked this question as dup of that Q/A, and to sum up: you can't pickle functions, and that's what you're trying to do when you pass wrapper() to Pool().map(). To pickle functions, you need to use copy_reg, as shown by this example.

" Classes, functions, and methods cannot be pickled -- if you pickle an object, the object's class is not pickled, just a string that identifies what class it belongs to. " (cf doc)

I don't use custom classes, and that's where his issue is (at least as far as I can tell)

no, his problem that he was trying to pickle an instance method, which is relatively close to a function, as both can't be pickled. And the workaround for that A should work for you too.

Though I did not test it..

vvvvv
  • 25,404
  • 19
  • 49
  • 81
zmo
  • 24,463
  • 4
  • 54
  • 90
  • 1
    you can pickle *top-level* functions; all multiprocessing has to do is pickle a string representing the name. I'm pretty sure the pickling error comes from trying to pickle `Both_Channels`, not `wrapper`. A bit hard to tell since the traceback is unhelpful. – roippi Mar 03 '14 at 21:55
  • 1
    or actually the pickling error probably comes from trying to ship `s.Noise_Reduction` objects back through the pipe. Bet that's it. – roippi Mar 03 '14 at 21:58
  • my feeling is that it's indeed wrapper that is failing (given the exception raised), though you may be right about `s.Noise_Reduction` being the thing within the function that makes it unpickable. – zmo Mar 03 '14 at 22:54
  • Is there an easy way to pickle several functions? Noise reductions also calls several other methods to reduce the noise in my signal. Am I just simply barking up the wrong tree to make things more efficient? – Cate Daniel Mar 03 '14 at 23:15
  • Well, I think, that whatever the real bug is, your problem is that you're trying to move around algorithms between processes instead of moving around only the datasets you want the algorithms to work on. Because the algorithm do not change for the life of the program, you can instanciate easily your processes with them, and get only the data to be pickled between them. – zmo Mar 03 '14 at 23:39
0

I ran into a simillaer error - And it was solved by changing the script running parameters

When I used pycharm run in a console option it failed like in the example with pickling Error lookup issue can't find function attribute.

While I used the run or debug mode it worked just find - I'm guessing it's loading the function diffrently on both ways.

Lihi
  • 1