is there andy synchronization problem in multiprocessing of python?

Question

The program is below, I use the function imap_unorder in the package multiprocessing for python

from multiprocessing import Pool

def f(x):
    return x,x*x

x_list = []
y_list = []
with Pool(processes=2) as pool:
    for x,y in pool.imap_unordered(f, range(4)):
        x_list.append(x)
        y_list.append(y)

Is the x_list and y_list keeping consistent ？

i know that the funciton imap_unordered doesn't process the input iterator orderly. but when outputing x and y, can them appended to the list at the same time?

x_list = [0,3,1,2]and y_list = [0,9,1,4] is a right example

but i don't want to output such as x_list = [0,3,1,2]and y_list = [0,1,9,4]

thanks a lot

Answer 1

The pairs will be internally consistent, but with imap_unordered() you're not guaranteed they're in the same order as the input iterable.

You could also use zip() to transpose a list-of-pairs into a pair-of-lists:

with Pool(processes=4) as pool:
    pairs = pool.imap_unordered(f, range(10))
    x_list, y_list = zip(*pairs)

Answer 2

The x,y values will always be consistent because you're back in the main process when you combine the results of f() into x_list and y_list.

To make this clearer, make the f() calls take varying times, like this:

from multiprocessing import Pool
import random
import time

def f(x):
    print( f"{x=}" )
    delay = 2.0*random.random()
    print( f"{delay=}" )
    time.sleep(delay)
    return x,x*x

x_list = []
y_list = []

if __name__ == "__main__":
    with Pool() as pool:
        for x,y in pool.imap_unordered(f, range(10)):
            x_list.append(x)
            y_list.append(y)
    print( f"{x_list=}" )
    print( f"{y_list=}" )

And you might get output like this, noting how concurrent execution of f() is causing interleaving of its print statements with other processes also executing their f():

x=0
delay=1.2836811458870878
x=1
delay=0.944416583067992
x=2
x=3
x=4
delay=1.6181526174149976
delay=0.5554265852002582
delay=1.5824716636923408
x=5
delay=0.01722248285756467
x=6
delay=1.8688128104508799
x=7
delay=0.9426899102018951
x=8
delay=0.5319453505012279
x=9
delay=0.24791521653497783
x_list=[5, 3, 8, 9, 1, 7, 0, 4, 2, 6]
y_list=[25, 9, 64, 81, 1, 49, 0, 16, 4, 36]

Examining the print output the x=5 has shortest time to sleep so is first in the results list. Similar x=6 has the longest sleep and so is last in the output.