Multiprocessing print statement doesn't work in Spyder IDE

Question

On a machine running Windows Server 2012 R2, in the Spyder IDE from Anaconda and running Python 3.7 with the following code:

import time
import multiprocessing

start=time.perf_counter()

def do_something():
    print('func start')
    time.sleep(1)
    print('func end')

if __name__=='__main__':
    print('name is main')
    p1=multiprocessing.Process(target=do_something)
    p1.start()
    p1.join()

finish=time.perf_counter()

print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')

And I get the output

name is main

Program completed in 0.13s

Goodbye!

My expectation was that I would see the two print statements

func start

func end

and also (because .join was envoked) that the program would take >1s to complete.

I suspect that the .start() call did not successfully call the do_something function.

FYI, I am following this tutorial, which I know needs to be modified to include the if statement on windows. I am also seeing similar code on other sites, but doesn't seem to work on my end.

Any suggestions on troubleshooting would be much appreciated.

**EDIT: Per the comment below from Azy_Crw4282, the code seems to work on his end and, per his suggestion, it seems to work from the cmd prompt. So this seems to be a bug specifically with the Spyder IDE.

FYI, I wanted to understand whether the issue was that the process was getting kicked off but the IDE wasn't capturing the output OR the process wasn't getting kicked off. I tried two things, 1) the code below writes a dataframe to csv. When doing this in the multiprocessing function, it does NOT write the file. 2) I created a global variable and changed variable value in the function. Spyder keeps the variable values after the code runs, and when I printed the variable it was unchanged.

So, in summary - it seems that the Spyder IDE does not work with the multiprocessing module.**

import time
import multiprocessing

start=time.perf_counter()

df=pd.DataFrame(data={'Col1':[1.1,2.1,3.1],
    'Col2':[1.2,2.2,3.2],
    'Col3':[1.3,2.3,3.3]}, columns=['Col1','Col2','Col3'])

def do_something():
    print('func start')
    df.to_csv('C:/testMp.csv')
    time.sleep(1)
    print('func end')



if __name__=='__main__':
    print('name is main')
    p1=multiprocessing.Process(target=do_something)
    p1.start()    
    p1.join()

    finish=time.perf_counter()

    print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
    print('Goodbye!')

Observation - can't run this in PyCrust (a wx REPL) but it runs perfectly in PyDev/Eclipse. Also runs from the OS bash shell. — Todd, Feb 23 '20 at 21:43
This reminds me of a project I had that made use of subinterpreters. Most modules work fine in that scenario, but some don't work right when loaded. I think these Python console apps like PyCrust may use subinterpreters (and some don't). That could explain why the multiprocessing module is behaving differently in the different shell environments. — Todd, Feb 23 '20 at 21:53
This will happen with any IDE where the default multiprocessing start method is "spawn", and the IDE uses STDOUT redirection to integrate the python instance into their gui. Spyder on windows does exactly this by default, but can easily be changed to use an external system terminal via "preferences > run > execute in a system terminal" — Aaron, Jul 13 '21 at 00:01

AzyCrw4282 · Accepted Answer · 2020-02-23T20:37:16.117

When I ran your code, I get the following output. Can you try running your code in another ide/ cmd line/ terminal?

import multiprocessing

start=time.perf_counter()

def do_something():
    print('func start')
    time.sleep(1)
    print('func end')

if __name__=='__main__':
    print('name is main')
    p1=multiprocessing.Process(target=do_something)
    p1.start()
    p1.join()

finish=time.perf_counter()

print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')

Outputs:

name is main


Program completed in 0.0s
Goodbye!
func start
func end


Program completed in 1.27s
Goodbye!

The above result is not probably what you expect. In order to achieve, what you want, you need to indent the outer print section so that it appears within the main call.

import time
import multiprocessing

start=time.perf_counter()

def do_something():
    print('func start')
    time.sleep(1)
    print('func end')

if __name__=='__main__':
    print('name is main')
    p1=multiprocessing.Process(target=do_something)
    p1.start()
    p1.join()

    finish=time.perf_counter()

    print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
    print('Goodbye!')

Outputs:

name is main
func start
func end


Program completed in 1.33s
Goodbye!

Thanks Azy. From the cmd prompt, I get the results you laid out. I guess this is an error in Spyder?? Also upvoted your reply but I dont have enough rep to change your post score. Thanks again. — DanS., Feb 23 '20 at 20:34
Yes, I would think so. I ran it in PyCharm and this worked perfectly. — AzyCrw4282, Feb 23 '20 at 20:35

Multiprocessing print statement doesn't work in Spyder IDE

1 Answers1

Linked