How to capture stderr from Python mulitprocessing process

Question

How do you capture the stdout/stderr of a specific target of a process launched from Python's mulitprocessing.Process library, ideally without requiring special modifications to the target function?

e.g.

p = Process(target=myfunc)
p.daemon=True
p.start()
p.join()
print p.exitcode, p.stderr.getvalue()

I know how to override sys.stdout/sys.stderr with a custom file object and use multiprocessing.Queue to redirect IO between processes, but this requires modifying the target function to write its output to the Queue.

Is something like this available with Process? I'm assuming it is on some level, since I'm able to see the process's output in the terminal when running, even if I can't record it from the parent process.

If you want an error message have a look at http://stackoverflow.com/questions/13535680/python-debug-tools-for-multiprocessing/14327070#14327070 although that may not fit your use case. — User, Feb 08 '14 at 09:42

User · Answer 1 · 2014-02-08T19:19:00.117

This is no answer but a long comment.

I would create a wrapper function that replaces sys.stderr and writes it to a file/pipe and then after/while the process ran the stderr is read into the parent. Reason:

This seams difficult if you do not alter the function

Multiprocessing does not use subprocess. You subclass Process and set Process.Popen to replace multiprocessing.forking.Popen. Replacing multiprocessing.forking.Popen must implement different code for the platforms 'win32' and 'not win32' leading to two different subclasses OR use subprocess.

Modifying multiprocessing.forking.Popen to use subprocess should be your best try from my point of view.

In Process.start()

from .forking import Popen
self._popen = Popen(self) # also self.Popen

In multiprocessing.forking

class Popen(object):
    def __init__(self, ...):
        # ...
        # for windows
        hp, ht, pid, tid = _subprocess.CreateProcess(
            _python_exe, cmd, None, None, 1, 0, None, None, None
            )

Now having a look at subprocess:

(p2cread, p2cwrite,
     c2pread, c2pwrite,
     errread, errwrite) = self._get_handles(stdin, stdout, stderr)
self._execute_child(args, executable, preexec_fn, close_fds,
                            cwd, env, universal_newlines,
                            startupinfo, creationflags, shell,
                            p2cread, p2cwrite,
                            c2pread, c2pwrite,
                            errread, errwrite)
self.stdout = os.fdopen(c2pread, 'rU', bufsize)

How to capture stderr from Python mulitprocessing process

1 Answers1