capture stdout and stderr of process that runs an infinite loop

Question

I want to run a process that runs an infinite loop (for example, starting a database server) from a python script and capture stdout and stderr. I tried this, but p.communicate() never returns, apparently because the process needs to finish first.

from subprocess import Popen, PIPE, STDOUT
cmd = "python infinite_loop.py"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
print("the process is running")
stdout, stderr = p.communicate()
print(stdout)

I'd like to get the output in some kind of streaming form. For example, I might want to save every 100 characters to a new log file. How can I do it?

I feel like I've done this with [asyncio.subprocess](https://docs.python.org/3/library/asyncio-subprocess.html), but I unfortunately don't have a code sample for you. (Following [this example](https://docs.python.org/3/library/asyncio-subprocess.html#subprocess-using-streams) specifically). — jedwards, Jul 02 '18 at 10:01

jedwards · Answer 1 · 2018-07-02T10:48:39.143

Edit: Something closer to what you already had, as asyncio seems like overkill for a single coroutine:

import sys
from subprocess import Popen, PIPE, STDOUT

args = (sys.executable, '-u', 'test4.py')
cmd = ' '.join(args)
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, universal_newlines=True)
print("the process is running")

for line in iter(p.stdout.readline,''):
    line = line.rstrip()
    print(line)

Original:

I threw something together. The following uses asyncio.subprocess to read lines from a subprocess' output, and then do something with them (in this case, just print() them).

The subprocess is specified by args, and in my case is just running another python instance in unbuffered mode with the following script (test4.py):

import time
for _ in range(10):
    print(time.time(), flush=True)
    time.sleep(1)

I'm sleeping in the for loop so it's clear whether the lines are coming in individually or all at once when the program has finished. (If you don't believe me, you can change the for loop to while True:, which will never finish).

The "supervisor" script is:

import asyncio.subprocess
import sys


async def get_lines(args):
    proc = await asyncio.create_subprocess_exec(*args, stdout=asyncio.subprocess.PIPE)

    while proc.returncode is None:
        data = await proc.stdout.readline()
        if not data: break
        line = data.decode('ascii').rstrip()
        # Handle line (somehow)
        print(line)


if sys.platform == "win32":
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
else:
    loop = asyncio.get_event_loop()

args = (sys.executable, '-u', 'test4.py')
loop.run_until_complete(get_lines(args))
loop.close()

Note that async def is Python 3.5+, but you could use @asyncio.coroutine in 3.4.

capture stdout and stderr of process that runs an infinite loop

1 Answers1