2

I would like to parallilize some computation and store results in a (say) list. I am unable to do so using joblib (and Python 3), here is a minimal example:

import itertools
from joblib import Parallel, delayed

## Function to create an iterator
def get_iterator(target):
    return itertools.product( ["A", "B"], [ target ])


## Polulates the target list
def my_fun(pair):
    pair[1].append(pair[0]) 


print( "These are the contents of the iterator")
target = []
for pair in get_iterator(target):
    print(pair)
# ('A', [])
# ('B', [])  


print("Standard serial code populates the target list")
target = []
it = get_iterator(target)
for pair in it:
    my_fun(pair)
print( target )
# ['A', 'B']

print("Parallelizing does not populate the list")
target = []
it = get_iterator(target)
Parallel(n_jobs=2)(delayed(my_fun)(pair) for pair in it )
print(target)
# []

Solution: Use the fact that delayed returns a list upon completion https://stackoverflow.com/a/34894862/3670097

Yair Daon
  • 1,043
  • 2
  • 15
  • 27
  • 1
    Possible duplicate of [How to write to a shared variable in python joblib](https://stackoverflow.com/questions/46657885/how-to-write-to-a-shared-variable-in-python-joblib) – Yair Daon Sep 23 '18 at 09:58

0 Answers0