18

I have recently started to delve into multiprocessing, as I believe my code can be easily parallelized. Upon working through the tutorials, though, I encountered an issue: functions distributed in a pool do not seem to print.

Here's the culprit:

__spec__ = None # This line is required for Spyder and not part of the actual example

from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

if __name__ == '__main__':
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

The output I receive is the following:

main line 
module name: __main__ 
parent process: 10812 
process id: 11348*

Now it is clear that the console only seems to print the info function, but not any output of the f function (which is using multiprocessing.Process). I have encountered similar issues with other examples I found online: computations are done and returned correctly when using multiprocessing, but prints never show up in the console.

Does anybody know why, and how to address this issue?

On a possibly related note, I am using Python 3.6 in Spyder 3.2.4 . Spyder seems to have a few quirks, as the first line in the code already is a workaround required to allow multiprocessing to work at all, an issue I found already discussed here. A similar, unresolved issue was mentioned here.

I would appreciate any help, and a happy new year to everyone.

Hai Vu
  • 37,849
  • 11
  • 66
  • 93
J.Galt
  • 377
  • 1
  • 2
  • 10
  • on linux it *does* print out the things in `f`. On which OS are you? – hansaplast Jan 03 '18 at 14:28
  • oh, and how do you start the script? I don't know spyder, but maybe you get different results when calling the script directly on the command line? – hansaplast Jan 03 '18 at 14:29
  • 3
    Multiprocessing doesn't work well on Windows in Spyder, sorry. You can run your code in an external terminal to have the results you want. To do that, please go to `Run > Configuration per file > Execute in an external system terminal`. – Carlos Cordoba Jan 03 '18 at 14:34
  • Hi, I am using Windows 10. In the IPython consolue in Spyder, the result is the same. In the Python console, I get an error message for the same script: AttributeError: 'module' object has no attribute 'getppid' – J.Galt Jan 03 '18 at 14:35
  • Hi Carlos, thanks, that does seem to work. Still, that's a bit unfortunate, since that seems to means the prints I require only show up in the external console. – J.Galt Jan 03 '18 at 14:37
  • I ran your script on both Windows 10 and Debian 8, both with Python 3.6 and got the expected output. I suspect that Spyder did something to the output. Try to run it from command line to see if it helps. – Hai Vu Jan 03 '18 at 18:33
  • Possible duplicate of [Simple Python Multiprocessing function doesn't output results](https://stackoverflow.com/questions/29629103/simple-python-multiprocessing-function-doesnt-output-results) – hansaplast Jan 03 '18 at 21:56
  • 3
    the solution is explained [here](https://stackoverflow.com/questions/29629103/simple-python-multiprocessing-function-doesnt-output-results): Spyder seems to redirect stdout and on linux, a forked process inherits stdout from the parent process, in windows that doesn't seem to be the case and the output is done on "default stdout" and hence doesn't land in the spyder console – hansaplast Jan 03 '18 at 21:58
  • 1
    Thank you very much! hansaplast's link seems to explain the root of the issue, and Carlos Cordoba's suggestion showed a potential workaround. – J.Galt Jan 04 '18 at 09:30
  • @CarlosCordoba: can you write your solution into an answer? I was looking for this option and couldn't find it – hansaplast Jan 04 '18 at 13:35

3 Answers3

34

(Spyder maintainer here) Multiprocessing doesn't work well on Windows in Spyder's IPython console. However, you can run your code in an external terminal to have the results you want.

To do that, please go to

Run > Configuration per file > Execute in an external system terminal.

Update: Since our 5.2.0 version (released in November 2021), prints generated while running multiprocessing code are captured and displayed in the IPython console for all operating systems.

Carlos Cordoba
  • 33,273
  • 10
  • 95
  • 124
  • 1
    Hi Carlos, thanks for this lifesaver. Is this fixed in the most current version of Spyder? When you say that multiprocessing does not work well do you mean it can create bugs when used with Spyder? or this is simply a printing issue that is solved using your trick? thanks!! – ℕʘʘḆḽḘ Apr 14 '18 at 19:46
  • 6
    That printing with multiprocessing on Windows doesn't work as expected. – Carlos Cordoba Apr 14 '18 at 20:35
  • 1
    any news why this happens? not fixed in v4.0? – user50473 Dec 13 '19 at 10:06
  • does not seem to be fixed in version 4 :/ – henry Apr 05 '20 at 07:30
  • How can one debug a code by running it in such a manner (in external terminal) ? – Debvrat Varshney Jun 10 '21 at 01:59
  • I didn't know about this issue so I put messages into a multiprocessing queue and had a thread outside the Process listen for items entering the queue and then print them. Printing isn't the only multiprocessing thing that doesn't work well in the Spyder iPython console on Windows. I was having erratic results connecting to USB cameras and webcam within a Process. The external terminal solved both of these issues. – slalomchip Feb 18 '22 at 17:48
  • This is regarding update. I am on Spyder 5.4.3. multiprocessing still does not work. I also do not see IPython console (It this regular Spyder console? Does I stand for interactive?) – user1700890 Jul 06 '23 at 16:23
  • 1
    @user1700890, sorry to hear that. Please open an issue in our [issue tracker](https://github.com/spyder-ide/spyder/issues) with a code example that doesn't work for you and we'll take a look at it. About the IPython console, I'm talking about [this pane](http://docs.spyder-ide.org/current/panes/ipythonconsole.html). – Carlos Cordoba Jul 06 '23 at 19:52
1

You can run it through Spyders' IPython console by saving the function as a different .py file and importing it into the script you're running. For example, save:

def f(name):
    info('function f')
    print('hello', name)

In a file called worker.py. Then in your main file, do the following: 

from multiprocessing import Process
import os
import worker

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

if __name__ == '__main__':
    info('main line')
    p = Process(target=worker.f, args=('bob',))
    p.start()
    p.join()
HollowedOkra
  • 11
  • 1
  • This doesn't work on my system. Also, I believe that the worker.py file needs the "info" function defined/imported. – mherzog Apr 19 '20 at 13:54
0

you can use log file instead. using fp.write()

Prabhakaran Thirugnanam
  • 11
  • 1