Simple Python Multiprocessing function in Spyder doesn't output results

Question

I have this very simple function right here in which I'm trying to run and test on, however, it doesn't output anything and it doesn't have any errors either. I've checked the code multiple times but it doesn't have any errors.

I printed jobs and here's what I got:

[<Process(Process-12, stopped[1])>, 
<Process(Process-13, stopped[1])>,
<Process(Process-14, stopped[1])>, 
<Process(Process-15, stopped[1])>,
<Process(Process-16, stopped[1])>]

Here's the code:

import multiprocessing

def worker(num):
    print "worker ", num
    return

jobs = []
for i in range(5):
    p = multiprocessing.Process(target = worker, args = (i,))
    jobs.append(p)
    p.start()

Here's the result I'm expecting but it's not outputting anything:

Worker: 0
Worker: 1
Worker: 2
Worker: 3
Worker: 4

@igor Thanks for checking. I'm using Spyder IDE from Anaconda to run the code. — OpenCv, Apr 14 '15 at 14:00
Do you simply want to `print` the results - which works in your snippet - or do you want to `return` them? — SmCaterpillar, Apr 14 '15 at 14:01
@SmCaterpillar I'm calling the function as a process, I just want to see the results. Like the 2 people above said the code works for them but it doesn't show on mine. — OpenCv, Apr 14 '15 at 14:04
Here's how I run it: $ python multiproc.so.py worker 0 worker 1 worker 3 worker 2 worker 4 — Igor, Apr 14 '15 at 14:05
Which OS? On windows it's very important to have a `if __name__ == '__main__':` guard before the relevant code parts (and on other OSes it's recommended too) — mata, Apr 14 '15 at 14:06
@igor thanks igor. you're running it on a python shell, which is probably why it's working. I'm using an IDE (Spyder) that i usually use and it's not showing anything. — OpenCv, Apr 14 '15 at 14:07
@mata yeah i'm on windows and still nothing. :( I tried that as well. — OpenCv, Apr 14 '15 at 14:09
Does Spyder redirect `stdout`? Windows does not support forking of processes. Accordingly, your new worker processes will print to the regular `stdout`, i.e. `sys.__stdout__`, and not to the redirected Spyder console because it won't inherit the redirection. — SmCaterpillar, Apr 14 '15 at 14:10
IPython notebook does redirect stdout. And this is definitely not forked. Anyhow, how about using logging instead of simple printing? — SmCaterpillar, Apr 14 '15 at 14:13
Yeah, as @SmCaterpillar has said, IDEs are notorious for not handling the stdout of background threads/processes well. — dano, Apr 14 '15 at 14:15
@SmCaterpillar would this help? adding this `sys.stdout.flush()` after the print statement. — OpenCv, Apr 14 '15 at 14:23
I guess not, the problem is that `sys.stdout` in your main process started by Spyder is different from `sys.stdout` in your new child processes. I would either switch to logging your output to files or to putting you results on a `queue` and collecting them in your main process. — SmCaterpillar, Apr 14 '15 at 14:47
@SmCaterpillar fantastic! would you like to make it an official answer so I can accept it. I think others will find this very helpful as well. — OpenCv, Apr 14 '15 at 15:19

SmCaterpillar · Accepted Answer · 2015-04-14T18:19:32.057

The comments revealed that OP uses Windows as well as Spyder. Since Spyder redirects stdout and Windows does not support forking, a new child process won't print into the Spyder console. This is simply due to the fact that stdout of the new child process is Python's vanilla stdout, which can also be found in sys.__stdout__.

There are two alternatives:

Using the logging module. This would encompass creating and logging all messages to one or several files. Using a single log-file may lead to the problem that the output is slightly garbled since the processes would write concurrently to the file. Using a single file per process could solve this.
Not using print within the child processes, but simply returning the result to the main process. Either by using a queue (or multiprocessing.Manager().Queue() since forking is not possible) or more simply by relying on the multiprocessing Pool's map functionality, see example below.

Multiprocessing example with a Pool:

import multiprocessing

def worker(num):
    """Returns the string of interest"""
    return "worker %d" % num

def main():
    pool = multiprocessing.Pool(4)
    results = pool.map(worker, range(10))

    pool.close()
    pool.join()

    for result in results:
        # prints the result string in the main process
        print(result)

if __name__ == '__main__':
    # Better protect your main function when you use multiprocessing
    main()

which prints (in the main process)

worker 0
worker 1
worker 2
worker 3
worker 4
worker 5
worker 6
worker 7
worker 8
worker 9

EDIT: If you are to impatient to wait for the map function to finish, you can immediately print your results by using imap_unordered and slightly changing the order of the commands:

def main():
    pool = multiprocessing.Pool(4)
    results = pool.imap_unordered(worker, range(10))

    for result in results:
        # prints the result string in the main process as soon as say are ready
        # but results are now no longer in order!
        print(result)

    # The pool should join after printing all results
    pool.close()
    pool.join()

your answer is correct and thanks. However just letting you know the program has been running for a while (few minutes now) with no output. It increased my CPU usage to 70% and hasn't stopped yet unless i manually kill the kernel. — OpenCv, Apr 14 '15 at 16:59
You can also try to use `imap_unordered` and immediately print the results as soon as they are ready, see my editing.Thereby, you may also spot the jobs that do not finish and cause the infinite loop. — SmCaterpillar, Apr 14 '15 at 18:16
thanks for this really nice question. is this still valid today? and if I understand well this is only a printing issue right? if I were interested only the the final output, then the Spyder issue you mention do not matter. Is that right? — ℕʘʘḆḽḘ, Apr 14 '18 at 19:59

Simple Python Multiprocessing function in Spyder doesn't output results

1 Answers1

Linked

Related