How to read the first byte of a subprocess's stdout and then discard the rest in Python?

Question

I'd like to read the first byte of a subprocess' stdout to know that it has started running. After that I'd like to discard all further output, so that I don't have to worry about the buffer.

What is the best way to do this?

Clarification: I'd like the subprocess to continue running alongside my program, I don't want to wait for it to terminate or anything like that. Ideally there would be some simple way to do this, without resorting to threading, forking or multiprocessing.

If I ignore the output stream, or .close() it, it causes errors if it is sent more data than it can fit in its buffer.

Does my answer not do exactly what you want? Launches subprocess, reads the first byte, then continues running the process in parallel... — Jesse Aldridge, Apr 01 '11 at 04:12
As far as I can tell, it would continue to read `stdout` and `stderr` into a buffer, wasting memory. I'm not sure, but this could also result in the subprocess blocking if the buffer fills up. I would like to avoid this. — Jeremy, Apr 01 '11 at 04:18
his solution won't block, since it never reads anything past the 1st byte. but you can't tell when it's done. `wait` will not read anything, but might block (if the program writes too much and starts waiting for the OS to read the pipe). my `communicate` won't, and it will read the contents, taking up memory. but you cant know when the process ends without reading... let me offer another solution — Claudiu, Apr 01 '11 at 04:28
@Jeremy I modified my example to prove that no memory is wasted. Run and watch the memory usage of the process with your process monitor. It never goes up. — Jesse Aldridge, Apr 01 '11 at 10:07
I have tested it, and the second process blocks once `stdout` buffer in the first process is full. — Jeremy, Apr 02 '11 at 02:14
Ah, so it does. +1 for working code. Btw, you've got an stdout where an stdin should be on line 11 :p — Jesse Aldridge, Apr 02 '11 at 08:41
If you want to know that the subprocess has "started running", or if it is still running, you do not need to read a byte of output. If you need some signal from the process that it has reached a certain state, modify that process to provide the indicator: have it create a file, or send a signal, or make a tcp connection, or send a message, or something reasonable. If you really want to check output, probably the simplest thing would be to connect the subprocess' stdout to a file, wait until that file gets a character written to it, then unlink it. This is not robust, but it's easy. — William Pursell, Aug 18 '16 at 14:07

score 67 · Answer 1 · edited Jul 29 '15 at 09:37

67

If you're using Python 3.3+, you can use the DEVNULL special value for stdout and stderr to discard subprocess output.

from subprocess import Popen, DEVNULL

process = Popen(["mycmd", "myarg"], stdout=DEVNULL, stderr=DEVNULL)

Or if you're using Python 2.4+, you can simulate this with:

import os
from subprocess import Popen

DEVNULL = open(os.devnull, 'wb')
process = Popen(["mycmd", "myarg"], stdout=DEVNULL, stderr=DEVNULL)

However this doesn't give you the opportunity to read the first byte of stdout.

edited Jul 29 '15 at 09:37

ctrl-alt-delor

7,506
5
40
52

answered Oct 17 '12 at 02:21

David Foster

6,931
4
41
42

8

In other words...it doesn't answer the question. – nneonneo Oct 17 '12 at 02:28
29

It answers the question in the title of the post. Therefore I think it would be useful to others who reach this page in an internet search. – David Foster Oct 18 '12 at 03:09
Does this interfere with `subprocess.check_call`? I wouldn't think so, since exit status is given by `wait`, however, just to be sure. – Patrick McLaren Aug 27 '13 at 03:58
1

This technique also works with check_call because it takes the same parameters as Popen. – David Foster Aug 31 '13 at 18:03
`DEVNULL` did not work for me. I am using Python 3.2.3 – ctrl-alt-delor Jan 05 '15 at 18:07
@richard needs python 3.3+... doesn't work for me in 3.4 anyway :/ – Wilf Jul 28 '15 at 19:35
@Wilf Did you say it works for all > 3.3, and does not work for 3.4 ? – ctrl-alt-delor Jul 29 '15 at 09:37
@richard its new in 3.3, so presumably it is should work in most builds of 3.3 and later (just not my build of python3 which may be missing bits, or I am doing something wrong :) . I used the `DEVNULL = open(os.devnull, 'wb')` as a workaround anyway. – Wilf Jul 29 '15 at 10:19
Works for me in 3.4.1. – David Foster Jul 30 '15 at 01:25
Now the title is edited, but this one is the answer for [python - How to hide output of subprocess - Stack Overflow](https://stackoverflow.com/questions/11269575/how-to-hide-output-of-subprocess) . (okay this one is posted very long ago) – user202729 Dec 05 '21 at 16:51

Jeremy · Answer 2 · 2011-04-02T19:41:20.760

This seems to work, but it doesn't feel idiomatic.

#!/usr/bin/env python3.1
import threading
import subprocess

def discard_stream_while_running(stream, process):
    while process.poll() is None:
        stream.read(1024)

def discard_subprocess_pipes(process, out=True, err=True, in_=True):
    if out and process.stdout is not None and not process.stdout.closed:
        t = threading.Thread(target=discard_stream_while_running, args=(process.stdout, process))
        t.start()

    if err and process.stderr is not None and not process.stderr.closed:
        u = threading.Thread(target=discard_stream_while_running, args=(process.stderr, process))
        u.start()

    if in_ and process.stdin is not None and not process.stdin.closed:
        process.stdin.close()

Example/test usage

if __name__ == "__main__":
    import tempfile
    import textwrap
    import time

    with tempfile.NamedTemporaryFile("w+t", prefix="example-", suffix=".py") as f:
        f.write(textwrap.dedent("""
            import sys
            import time

            sys.stderr.write("{} byte(s) read through stdin.\\n"
                             .format(len(sys.stdin.read())))

            # Push a couple of MB/s to stdout, messages to stderr.
            while True:
                sys.stdout.write("Hello Parent\\n" * 1000000)
                sys.stderr.write("Subprocess Writing Data\\n")
                time.sleep(0.5)
        """))
        f.flush()

        p = subprocess.Popen(["python3.1", f.name],
                             stdout=subprocess.PIPE,
                             stdin=subprocess.PIPE)

        p.stdin.write("Hello Child\n".encode())

        discard_subprocess_pipes(p) # <-- Here

        for s in range(16, 0, -1):
            print("Main Process Running For", s, "More Seconds")
            time.sleep(1)

you are not waiting for the first byte to be read. Though: `p.poll() is None` after `Popen()` means that the subprocess *is* running. If it is all you need then you should [use `DEVNULL`-based solution instead](http://stackoverflow.com/a/12926181/4279). — jfs, May 16 '15 at 16:55

How to read the first byte of a subprocess's stdout and then discard the rest in Python?

2 Answers2

Example/test usage

Linked

Related