Send subprocess.Popen stdout, stderr to logging module

Question

I want to start an application using Pythons subprocess.Popen and send the output of the applications stdout and stderr to the logging module in such fashion that it shows like logging.INFO("This is whats in stdout") every time the applications sends something/someline to stdout/stderr.

Since the application is a deamon it does ( and should ) not terminate.

Is there an easy way to achieve this or do I have to check the process output frequently with a separate thread?

score 5 · Answer 1 · edited May 23 '17 at 12:09

Here's what I took from j-f-sebastian's answer as a reusable class:

import subprocess
from threading import Thread

class BackgroundPopen(subprocess.Popen):
    @staticmethod
    def prefix_handler(prefix, io):
        return lambda line: io.write(prefix + line)

    @staticmethod
    def _proxy_lines(pipe, handler):
        with pipe:
            for line in pipe:
                handler(line)

    def __init__(self, out_handler, err_handler, *args, **kwargs):
        kwargs['stdout'] = subprocess.PIPE
        kwargs['stderr'] = subprocess.PIPE
        super(self.__class__, self).__init__(*args, **kwargs)
        Thread(target=self._proxy_lines, args=[self.stdout, out_handler]).start()
        Thread(target=self._proxy_lines, args=[self.stderr, err_handler]).start()

In this example, `prefix_handler` is an example of a possible `out_handler` and `err_handler`; it does not override anything in `subprocess.Popen` and can be safely omitted. — artoonie, Sep 08 '17 at 01:16

score 4 · Answer 2 · edited Jun 20 '17 at 09:16

A thread is a simple and portable way to consume the output (not tested):

#!/usr/bin/env python
import logging
from subprocess Popen, PIPE, STDOUT
from threading import Thread

def consume_lines(pipe, consume):
    with pipe:
        for line in iter(pipe.readline, b''): #NOTE: workaround read-ahead bug
            consume(line)

logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
consume = lambda line: logging.info('this is whats in the output: %r', line)

process = Popen(command, stdout=PIPE, stderr=STDOUT, bufsize=1)
Thread(target=consume_lines, args=[process.stdout, consume]).start()
process.wait()

score 3 · Answer 3 · answered Feb 19 '16 at 08:51

3

In my experience, a separate thread is the way to go. That's how my sarge library does it - it uses threads under the hood to capture output from the child process. In fact, I generally use two threads (one for stdout and one for stderr) unless I am merging the two output streams in the subprocess.Popen call.

answered Feb 19 '16 at 08:51

Vinay Sajip

95,872
14
179
191

This answer would be better with a code example (I know there is one in the linked page). – Superbest Jan 26 '18 at 23:35
There's also an alternative method using non-blocking streams and just capturing it in the main thread while it waits for the child process to complete: https://stackoverflow.com/a/9115366/582917 What do you think of that vs the threads idea? – CMCDragonkai Mar 20 '19 at 03:30
@CMCDragonkai that example uses `select`, so it's not ideal for Windows. – Vinay Sajip Mar 21 '19 at 07:50

Send subprocess.Popen stdout, stderr to logging module

3 Answers3