Mutual list in multiple processes

Question

Let's say that I have an empty list my_list and 5 processes pr1, pr2, pr3, pr4, pr5, where each of them appends something to my list.

My question is : Is something like this possible? And will it behave normaly or an error will occur?

Answer 1

If these processes are independent then something like this is not possible. At least not without some additional mechanisms like sockets (which greatly increases complexity).

If these are created via multiprocessing and my_list is just a list then each process will have its own copy of the list.

Again in multiprocessing if you define my_list as multiprocessing.Queue then indeed this will be a shared structure between processes.

Answer 2

Are all 5 of your processes applying the same function, but to different inputs? If this is the case, then using Pool.map() from multiprocessing is appropriate:

from multiprocessing import Pool

n_processes = 5 #number of processes to use
input = [0.,1.,2.,3.] #some input

def f(x):
    #the function being applied on each thread
    return x**2. #for example

p = Pool(n_processes)
my_list = p.map(f,input)
p.close() #close the pool now you're finished with it

So the function f(x) is being applied to different inputs on each process, and then the output is collected in a list (equivalent to your 'my_list' that was empty to begin with). The output array is ordered in the same way as the input array i.e. if input = [x1,x2,x3,x4,x5] then output=[f(x1),f(x2),f(x3),f(x4),f(x5)].