multiprocessing pool map returning the same values for each call

Question

I'm having a problem getting different return values from a pool.map call. Im running this simplified example on a macbook, in a jupyter notebook with python 3.9:

import multiprocessing as mp
import numpy as np

def gen_random_list(num):
    print(f"Num: {num}\n")
    ret_list = np.random.randint(0, 5, size=3).tolist()
    return ret_list
    
with mp.Pool() as pool:
    results = []
    results = pool.map(gen_random_list, range(4))
    print(results)

I keep getting the same list back for each call to gen_random_list. The complete output is this:

Num: 0
Num: 1
Num: 2
Num: 3




[[0, 3, 2], [0, 3, 2], [0, 3, 2], [0, 3, 2]]

Does anyone know why every call to gen_random_list is returning the same values (in this run: [0, 3, 2])? And how do I fix this so I get a different random list back each time?

Thanks Brett

The code as written does not run unless I put the multiprocessing part inside of `if __name__ == '__main__'`. Once I do that, I cannot reproduce your results, I just got `[[0, 1, 2], [2, 0, 2], [2, 4, 3], [2, 1, 0]]`. — Kraigolas, Aug 09 '22 at 15:38
It works as written in a jupyter notebook. The check for main, won't work in a notebook, since the notebook isn't the main process. — Brett Elliot, Aug 09 '22 at 15:54
This is probably because the processes use the same seed to generate the numbers. Relevant https://stackoverflow.com/questions/73214631/pythons-multiprocessing-pool-does-unwanted-caching?noredirect=1&lq=1 — Charchit Agarwal, Aug 09 '22 at 17:13

score 0 · Answer 1 · answered Aug 09 '22 at 17:39

Thanks @Charchit! That was it! Subtle for sure.... Here's a working example now:

import multiprocessing as mp
import numpy as np

def gen_random_list2(num):
    print(f"Num: {num}\n")
    # https://stackoverflow.com/questions/22994423/difference-between-np-random-seed-and-np-random-randomstate
    local_state = np.random.RandomState(num)
    ret_list = local_state.randint(0, 5, size=3).tolist()
    return ret_list
    
with mp.Pool(processes=processes) as pool:
    results = list(map(gen_random_list2, range(100)))
    print(results)

multiprocessing pool map returning the same values for each call

1 Answers1