0

This is a follow-up question for the question I asked here. I tried to parallelize my code as follows:

import concurrent.futures as futures

class A(object):

    def __init__(self, q):
        self.p = q

    def add(self, num):
        r = 0
        for _ in xrange(10000):
            r += num
        return r


num_instances = 5

instances = []
for i in xrange(num_instances):
    instances.append(A(i))

n = 20
# Create a pool of processes. By default, one is created for each CPU in your machine.
results = []
pool = futures.ProcessPoolExecutor(max_workers=num_instances)

for inst in instances:
    future = pool.submit(inst.add, n)
    results.append(future.result())

pool.join()

print(results)

But, I got this error:

Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 268, in _feed send(obj) PicklingError: Can't pickle : attribute lookup builtin.instancemethod failed

Any idea why I get this error? I know we can use map function to assign jobs, but I intentionally don't want to do that.

user491626
  • 357
  • 1
  • 2
  • 14
  • `multiprocessing` (what `concurrent.futures.process` is based on) doesn't share memory natively so you cannot (normally, there are exceptions) have an object instance (or its methods) called from multiple processes. Read [`this answer`](https://stackoverflow.com/a/44186168) for more info and a potential workaround. – zwer Jul 18 '18 at 21:44
  • @zwer I am not calling one object instance from multiple processes. I am calling different object instances. "instances" is a list of object instances, I am calling them one by one and assign each one to one process. – user491626 Jul 18 '18 at 22:02
  • 1
    Those instances were created in your main process, you cannot just _pass_ them to another process - when you start a new process it has no idea there are some instances in another process which is why Python tries to _pickle_ the instance to send it to the new process... and fails as you cannot pickle an instance method. – zwer Jul 18 '18 at 22:19
  • @zwer wow, you are right. So, when I create instances, I have to assign them to different processes right there? How can I do that? – user491626 Jul 18 '18 at 22:36
  • Create the instances in the processes themselves and just pass enough information to instantiate them there. The other alternative, which I do not recommend, is to have it done semi-automatically as in the linked answer. – zwer Jul 18 '18 at 22:51
  • @zwer you mean I can just write pool.sumbit(A,i) to create an instance inside a process? – user491626 Jul 18 '18 at 23:19
  • @zwer I am confused. If what you are saying is correct, why does my code works here? https://stackoverflow.com/questions/51407464/why-doesnt-parallelizing-decrease-the-runtime There I assign object instances to different processes later (after creating the instances) – user491626 Jul 18 '18 at 23:46

0 Answers0