Parallelizing a python function that takes a dict as input

Question

I have the following function:

class BulkRemoteiTunesUpdater():

    def __init__(self):
        ...
    def run_update(self, update_dict):
        # do something

I want to parallelize this process. So far I have:

if __name__ == '__main__':
    b=BulkRemoteiTunesUpdater()
    b.fetch_updates_to_do_info()

    fetched_update_info = b.fetched_update_info
    pool = Pool(NUM_IN_PARALLEL)
    pool.map(b.run_update, fetched_update_info)
    pool.join()

What I have above raises the following error:

PicklingError: Can't pickle <type 'instancemethod'>: 
    attribute lookup __builtin__.instancemethod failed

What am I doing wrong here? And how would I correctly parallelize this?

fetched_update_info returns to me a list of dict objects that I need for my function `run_update`. — David542, Jun 30 '14 at 22:20

score 1 · Accepted Answer · edited May 23 '17 at 12:20

1

As found here, the problem is that your iterable instancemethod fetched_update_info is not picklable. You'll have to go through the hassle of making it so to parralelize it.

OR, if you can get a list out of the iterable first, you may be able to avoid this, such as:

pulled = []
for obj in b.fetched_updated_info():
    pulled.append(obj)

pool.map(b.run_update, pulled)

edited May 23 '17 at 12:20

Community

1
1

answered Jun 30 '14 at 22:25

Philip Massey

1,401
3
14
24

johntellsall · Answer 2 · 2014-06-30T22:23:36.790

0

two small changes:

if __name__ == '__main__':
    b=BulkRemoteiTunesUpdater()
    b.fetch_updates_to_do_info() # a list of dicts

    pool = Pool(NUM_IN_PARALLEL)
    pool.map(b.run_update, b.fetched_update_info()) # fetch data
    # not needed: pool.close()
    pool.join()

Here's a full example of how to use Pool.map:

source

import multiprocessing

def calc(num):
    return num*2

pool = multiprocessing.Pool(5)
for output in pool.map(calc, [1,2,3]):
    print 'output:',output

output

output: 2
output: 4
output: 6

edited Jun 30 '14 at 22:23

answered Jun 30 '14 at 22:16

johntellsall

14,394
4
46
40

This `b.fetched_update_info()` now returns the Error `list object is not callable`. – David542 Jun 30 '14 at 22:21

score 0 · Answer 3 · answered Jun 30 '14 at 22:56

0

I fixed this by extracting the instance methods and converting them into normal functions. Then when I called the function I didn't get the pickle errors.

answered Jun 30 '14 at 22:56

David542

104,438
178
489
842

Parallelizing a python function that takes a dict as input

3 Answers3

source

output