Python Multiprocessing Semaphore not working

Question

I want my program to print out one line at a time, however it's printing multiple at a time and creating a garbled mess. I cannot seem to find out why the semaphore is not preventing multiple processes from printing over each other.

How can I get it to respect the semaphore?

Here is a simplified version of my code that has the same problem when run it (I'm running on Windows with Python 2.7.11 (this can't be changed)):

import multiprocessing

rightofway = multiprocessing.Semaphore(1)

def writenum(number):
    rightofway.acquire()
    print("[+] - " + str(number))
    rightofway.release()
    return

def main():
    starting = 0
    ending = 50

    list = range(starting, ending)

    pool = multiprocessing.Pool(10)
    pool.map(writenum, list)
    return

#Required for Windows multiprocessing
if __name__ == '__main__':
    main()

Here is an example of garbled output:

[+] - 0
[+] - 1
[+] - 2
[+] - 3
[+] - 4
[+] - 5
[+] - 6
[+] - 7
[[+] - 8
+] - 10[
+] - 9[+] - 11
[+] - 12

[[+] - 13+] - 14

[[+] - 15+] - 16

[[+] - 18+] - 17

[[+] - 19+] - 20

[[+] - 22+] - 21

[[+] - 23+] - 24

[[+] - 26+] - 25

[[+] - 27+] - 28

[[+] - 30+] - 29

[[+] - 31+] - 32

[[+] - 34+] - 33

[[+] - 35+] - 36

[[+] - 38+] - 37

[[+] - 39+] - 40

[[+] - 42+] - 41

[[+] - 43+] - 44

[[+] - 46+] - 45

[[+] - 47+] - 48

[+] - 49

Here's an example of the output I want (note I don't care about the order):

[+] - 0
[+] - 1
[+] - 2
[+] - 3
[+] - 4
[+] - 5
[+] - 6
[+] - 7
[+] - 8
[+] - 9
[+] - 10
[+] - 11
[+] - 12
[+] - 13
[+] - 14
[+] - 15
[+] - 16
[+] - 17
[+] - 18
[+] - 19
[+] - 20
[+] - 21
[+] - 22
[+] - 23
[+] - 24
[+] - 25
[+] - 26
[+] - 27
[+] - 28
[+] - 29
[+] - 30
[+] - 31
[+] - 32
[+] - 33
[+] - 36
[+] - 34
[+] - 35
[+] - 37
[+] - 38
[+] - 40
[+] - 39
[+] - 41
[+] - 42
[+] - 44
[+] - 43
[+] - 45
[+] - 46
[+] - 48
[+] - 47
[+] - 49

I don't have windows handy, but this accepted solution uses an initializer with the pool. its my guess for the fix. a lock should work for you or adapt for semaphore https://stackoverflow.com/questions/28664720/how-to-create-global-lock-semaphore-with-multiprocessing-pool-in-python?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa — tdelaney, May 16 '18 at 20:07
@tdelaney - I attempted it and it resulted in exactly the same kind of garbled output. Lock isn't working either. — Danegraphics, May 16 '18 at 20:18

score 5 · Accepted Answer · edited Jun 20 '20 at 09:12

You question is similar to this one.

From the multiprocessing programming guidelines.

Explicitly pass resources to child processes

... it is better to pass the object as an argument to the constructor for the child process.

Apart from making the code (potentially) compatible with Windows ...

On Windows, you need to pass the shared objects to the Process constructor list of arguments. Otherwise, the child process will get a brand new copy instead of the parent's one. That's why you get the impression the Semaphore isn't working. The two processes are creating their own distinct Semaphore object instead of sharing the same one.

To pass a Semaphore object to a Pool on Windows you need to struggle a bit but not too much. As you cannot pass the Semaphore object to the writenum function directly, you need to rely on the Pool initializer.

from multiprocessing import Semaphore, Pool

mutex = None

def initializer(semaphore):
    """This function is run at the Pool startup. 
    Use it to set your Semaphore object in the child process.

    """
    global mutex

    mutex = semaphore

def writenum(args):
    with mutex:
        print "[+] - " + str(number)

def main():
    semaphore = Semaphore()
    pool = Pool(initializer=initializer, initargs=[semaphore])

    numbers = range(50)

    pool.map(writenum, numbers)

EDIT: just noticed I wrote about Lock instead of Semaphore. The core reasoning remains the same.

That worked! And I think I see why the code is that way, but lemme ask just in case. Why do we need to have mutex=None at the top even though it is declared in the initializer function? Also, why is the initializer function not a part of the Pool object? — Danegraphics, May 17 '18 at 12:53
Can you better explain what do you mean with the question `why is the initializer function not a part of the Pool object`? — noxdafox, May 17 '18 at 13:00
It appears that Pool() is built to take arguments and bring them into the global space of the processes. If so, then why do I have to write part of that functionality myself instead of simply passing the (already global?) semaphore to the processes? — Danegraphics, May 17 '18 at 13:09
The `Pool` has two ways of passing data to child processes. The first is via the `apply`/`map` function arguments. This requires arguments to be pickleable and `Semaphores` are not. You can only rely on the second way which is by using the `initializer` function. As that function cannot return values, only way for your data to persist is by using global variables. — noxdafox, May 18 '18 at 11:56

score 0 · Answer 2 · answered May 17 '18 at 08:00

To make things a bit easier, Following worked for me. Tested on Win10. TL;DR - Use locks not semaphore

import multiprocessing

rightofway = multiprocessing.Lock()

def writenum(number):

    with rightofway:
        print("[+] - " + str(number))

    return

def main():
    starting = 0
    ending = 50

    list = range(starting, ending)

    pool = multiprocessing.Pool(10)
    pool.map(writenum, list)
    return

#Required for Windows multiprocessing
if __name__ == '__main__':
    main()

This isn't working for me for some reason. Same jumbled results. — Danegraphics, May 17 '18 at 12:46

Python Multiprocessing Semaphore not working

2 Answers2