1

I would like to know in what scenarios python's multiprocessing libary would give the Can't pickle <type 'instancemethod'> error when using the pool.apply_async, but not when using the process.start method. If I understood correctly, the process.start method should also be pickling the code, so not much should differ in terms of requirements, or am I mistaken?

Here is the general structure of the code I am having this issue with (the actual functions are too long and complex to condense accurately):

import multiprocessing as mult

def f2(name):
    print 'hello' + name

def f1(x):
    return str(x)

if __name__ == '__main__':
    #This works fine
    processes = []
    for x in xrange(100):
        name = f1(x)
        p = mult.Process(target=f2, args=(name,))
        processes.append(p)
    for p in processes:
        p.start()

if __name__ == '__main__':
    #This gives the pickle error
    pool = mult.Pool() 
    for x in xrange(100,200):
        name = f1(x)
        pool.apply_async(f2, args=(name,))
    pool.close()

Note: the completed processes do not need to join with the main process after completing. Also, I am running this on Win7 64 with a 64bit version of Python.

I hope this question is different enough from this one.

Community
  • 1
  • 1
TimY
  • 5,256
  • 5
  • 44
  • 57
  • Are you on Windows? I'm on Ubuntu and get no error. – Reut Sharabani Dec 01 '14 at 09:43
  • @ReutSharabani - Yes I am. – TimY Dec 01 '14 at 09:46
  • 1
    @ReutSharabani - You wouldn't get an error with this specific code, I presume that the actual f1 and f2 functions (not the one in the snippet) are the cause of the error. – TimY Dec 01 '14 at 09:49
  • Well, I can't help you without knowing what you're trying to pickle. Where are the actual methods defined? In the global scope of the module? Here is a list of what can be pickled: https://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled – Reut Sharabani Dec 01 '14 at 09:52
  • 1
    I'm starting to think that the method I'm running is simply impossible to pickle (too long to check), but this doesn't explain why `process.start` works. Does process.start not need to pickle the method? – TimY Dec 01 '14 at 09:54
  • My guess is that the call to `Process` defines the data in the spawning process and then `fork`s. The call to `Pool`, on the other hand, `fork`s some processes and then defines the data (which is not available on the spawned processes later...) – Reut Sharabani Dec 01 '14 at 10:02
  • 1
    See [this question](http://stackoverflow.com/questions/27318290/why-can-i-invoke-a-multiprocessing-process-with-an-instance-method-as-the-target/27320254#27320254) for the reason for this and a solution. – dano Dec 05 '14 at 22:35
  • Thanks @dano - this is an excellent answer. A bit frustrating that no one answered when I posted this question. Also, not that it really matters, but technically-speaking, the one you are referring to is the duplicate. – TimY Dec 06 '14 at 08:31
  • @TimY I think the issue was that your question didn't have any code that could be used to reproduce the problem,so it wasn't completely clear what issue you were hitting. And yes, I did realize the other question was technically the dupe, but I had already written my answer there before I noticed this one in the related items tab. Had I saw it before I would have put the answer here. – dano Dec 06 '14 at 14:23
  • That's a good point. No problem. In any case, many thanks for linking! – TimY Dec 07 '14 at 09:41

0 Answers0