Trouble writing python output to file with multiprocessing pool

Question

I have found some similar questions to mine on this forum but none that has helped with my specific problem (I'm fairly new to python so forgive me if I should've understood the answer from a different thread). I am actually working with a fairly complicated python code, but to illustrate the problem I've having I have written a very minimal code that exhibits the same behavior:

    from multiprocessing import Pool

    log_output = open('log','w')

    def sum_square(number):
        s = 0
        for i in range(number):
            s += i * i
        log_output.write(str(s)+'\n')
        return s

    if __name__ == "__main__":
        numbers = range(5)
        p = Pool()
        result = p.map(sum_square,numbers)

        p.close()
        p.join()

When this code is executed, there are no errors or warnings, etc., but the resulting file "log" is always totally empty. This is the same behavior I find in my other more complicated python program also using a multiprocessing pool. I had expected that the processes would probably print to file in the wrong order, but I'm confused why nothing prints at all. If I get rid of the multiprocessing and just run the same basic code with a single process, it prints to file just fine. Can anyone help me understand what I'm doing wrong here? Thanks.

Does this answer your question? [Python multiprocessing safely writing to a file](https://stackoverflow.com/questions/13446445/python-multiprocessing-safely-writing-to-a-file) — Kroshka Kartoshka, Oct 29 '20 at 21:13

score 0 · Accepted Answer · answered Oct 29 '20 at 22:07

I think you're well into undefined territory here, but just flushing the file works for me. for example:

from multiprocessing import Pool

log_output = open('log', 'w')

def sum_square(number):
    print(number, file=log_output)
    log_output.flush()

if __name__ == "__main__":
    with Pool() as p:
        p.map(sum_square, range(5))

works for me in Python 3.8 under Linux. I've also shortened your code to clarify the issue, and make it slightly more idiomatic.

Trouble writing python output to file with multiprocessing pool

1 Answers1