How to fetch elements from a generator using Pool?

Asked Jul 30 '17 at 14:15

Active Jul 30 '17 at 14:57

Viewed 48 times

I have a generator that I want to listify.

gen = requests.get(url, stream=True).iter_content(chunk_size=100)

I want to listify this parallely.

out = list(gen) does it synchronously.

Running this:

from multiprocessing import Pool
out = Pool(10).imap((lambda x: x), gen)

gives error:

    raise value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

edited Jul 30 '17 at 14:57

asked Jul 30 '17 at 14:15

canonball

Replace your `lambda` with an actual named function and it should work. Alternatively, `import dill`. [More info on lambda pickling](https://stackoverflow.com/questions/25348532/can-python-pickle-lambda-functions) – jDo Jul 30 '17 at 15:08
It did, but why doesn't lambda work? – canonball Jul 30 '17 at 16:24

0 Answers0