Make processes output one at a time in python

Question

I'm using Pool under multiprocessing to do some stuff.

def my_func(...):
    #Different processes can take different time
    print a, b, c #Value that I calculated above. (includes 2 new-lines)

There have been instances where values of a, b, c for a process are not printed together. Is there a way to avoid that? Can anyone explain as to what's happening here and how it can be avoided?

My understanding says that if I remove all newlines in the print and keep only one at the end, it should fix the problem. (Problem is, it's not reproducible every time so I'm still testing stuff).

Is there a way I can take sys.stdout exclusively for a process and then release it while I'm printing stuff to STDOUT?

How about simply concatenate them first? Like `' '.join([a, b, c])` and then print the result string. — Igonato, Oct 01 '13 at 07:31
@lgonato, problem is values of a, b, and c containing newlines. This won't fix it, right? — shadyabhi, Oct 01 '13 at 07:55

score 5 · Accepted Answer · answered Oct 01 '13 at 07:47

You could use a multiprocessing.Lock to serialize the prints. Create the common lock in the main program and pass the same lock to all the child processes. Example:

#!python3
from multiprocessing import Process,Lock
import time
import sys

def test(n,lock):
    with lock:
        for i in range(20):
            print(n,end='')
            sys.stdout.flush()
            time.sleep(.01)  # needed some delay or ran too fast and didn't mix output.
        print()

if __name__ == '__main__':
    lock = Lock()
    jobs = [Process(target=test,args=(n,lock)) for n in range(5)]
    for job in jobs:
        job.start()
    for job in jobs:
        job.join()

With with lock: commented out:

003023120134201342013420314203140231402134203140231420134021342031402134201342013420314203142
1342
14

4

As written:

00000000000000000000
11111111111111111111
44444444444444444444
22222222222222222222
33333333333333333333

Thanks @Mark, I like it. Will accept answer once I do it in my code as I'm using the module a little differently. — shadyabhi, Oct 01 '13 at 08:01

score 0 · Answer 2 · edited May 23 '17 at 12:24

0

Maybe setting this variable before running your program will help you, it force python to flush to stdout after each write :

export PYTHONUNBUFFERED=1

Make processes output one at a time in python

2 Answers2