Python: Multithreaded list append does not work as expected

Question

Why is the following code (you can run it here):

from multiprocessing.pool import Pool
from time import sleep

MY_LIST = []


def worker(j):
    global MY_LIST
    sleep(1)
    MY_LIST.append(j)
    print(j, len(MY_LIST))


if __name__ == '__main__':
    parameters= [1, 2, 3, 4]
    with Pool(processes=2) as pool:
        results = pool.map(worker, parameters)

prints:

And if I comment out the sleep(1) part, it prints what I expect:

How can I fix the code so that it correctly appends and in the end I have 4 elements in MY_LIST

Curiously, this code won't execute for me. I get `AttributeError: Can't get attribute 'worker' on ` — Sam Morgan, Mar 30 '20 at 00:24
@SamMorgan try this: https://repl.it/@CP8/PhonyGlossyKnowledge — Caner, Mar 30 '20 at 00:26
What is your goal? You probably need to use some synchronization mechanism when it comes to have several workers in parallel. — dcg, Mar 30 '20 at 01:10
Looks like a timing issue. `pool.map` splits the iterable into chunks and submits these as individual tasks to the workers. My guess is that in the version without the `sleep` one of the workers quickly grabs all four tasks and executes them whereas the version with the `sleep` allows for both workers to pick up two of the tasks each. You can `print(multiprocessing.current_process().name, j, len(MY_LIST))` in the `worker` function to see if that is the case. — shmee, Mar 30 '20 at 02:59

score 1 · Answer 1 · answered Mar 30 '20 at 04:50

With mutliprocessing, you would need to use a shared variable in order to access it from multiple processes. In your code, without the sleep, each process finishes fast enough so that you only need to use one process in the pool and so your code appends to the same MY_LIST. When you use a sleep, it uses more than one process and so you end up appending to different instances of MY_LIST.

You will need to use a shared variable in order to access it from each process. The shared variable would need a preallocated size:

from multiprocessing import pool, Array, current_process

MY_LIST = Array('i', 8)

def worker(j):
    MY_LIST[j-1] = j*j
    print(current_process().name, j, len(MY_LIST))

if __name__ == '__main__':
    parameters= [1, 2, 3, 4, 5, 6, 7, 8]
    with pool.Pool(processes=4) as pool:
        results = pool.map(worker, parameters)
    for i in MY_LIST:
        print(i)

Output:

ForkPoolWorker-1 1 8
ForkPoolWorker-2 2 8
ForkPoolWorker-1 4 8
ForkPoolWorker-3 3 8
ForkPoolWorker-1 5 8
ForkPoolWorker-2 6 8
ForkPoolWorker-1 7 8
ForkPoolWorker-3 8 8
1
4
9
16
25
36
49
64

score 1 · Accepted Answer · answered Mar 30 '20 at 04:55

There is a race which is revealed by the call to time.sleep because without it one of the workers finishes the task before any of the rest can even start. This is made clear if you use a longer list, for example:

from multiprocessing.pool import Pool
from time import sleep

MY_LIST = []

def worker(j):
    global MY_LIST
    sleep(1)
    MY_LIST.append(j)
    print(j, len(MY_LIST))

if __name__ == '__main__':
    parameters = range(25)
    with Pool(processes=2) as pool:
        results = pool.map(worker, parameters)

Will output something like

Of course that's not the only problem. The other issue is that global variables are not shared between processes, at least not the primitive python types. You need to use a multiprocessing.Manager.List or similar, something like this:

from multiprocessing import Pool, Lock, Manager
from time import sleep

manager = Manager()
MY_LIST = manager.list()

def worker(j):
    global MY_LIST, lock
    MY_LIST.append(j)
    print(j, len(MY_LIST))


if __name__ == '__main__':
    parameters = range(25)
    with Pool(2) as pool:
        results = pool.map(worker, parameters)

Which will output

Of course this does nothing to guarantee order, only access.

Python: Multithreaded list append does not work as expected

2 Answers2