6
from joblib import Parallel, delayed

def func(v):
    temp.append(v)
    return

temp = []
Parallel(n_jobs=4)(delayed(func)(v) for v in range(10))
print temp

I want to make shared memory variable. But the value of temp is empty []. How can I do it?

For other method, I tried pickle.dump and load. But there is a lock problem. Please give me advice!

KyungHoon Kim
  • 2,859
  • 2
  • 23
  • 26

2 Answers2

11
from joblib import Parallel, delayed

def func(v):
    return v

temp = Parallel(n_jobs=4)(delayed(func)(v) for v in range(10))
print temp

delayed collects the output returned by func in a list and returns it on completion.

Chintak
  • 365
  • 4
  • 14
8

You need to use multiprocessing.Manager.list, for example:

from joblib import Parallel, delayed
from multiprocessing import Manager

manager = Manager()
temp = manager.list()

def func(v, temp):
    temp.append(v)
    return

_ = Parallel(n_jobs=4)(delayed(func)(v, temp) for v in range(10))

temp[:]:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Manualmsdos
  • 1,505
  • 3
  • 11
  • 22
  • I doesn't run like that anymore. Move `def func...` to after the imports, everything else inside `if "__name__" == __main__:`. Then import `freeze_support` and call it (it's a function) right after the `if` line. – Jann Poppinga Jun 28 '22 at 06:24
  • 1
    What do you mean @JannPoppinga by "it doesn't run like that anymore"? I've tried it and it works. Do you have a specific version of a library in mind? – Codoscope Feb 28 '23 at 11:15