Subprocess buffer overflow

Question

I use python to test an already compiled binary. The idea is that:

• I open the subprocess with a different program (this separate process does nothing and listens to the commands)
• Then I send various commands to this subprocess (using mysubprocess.stdin.write())
• Then (depending on my need) I validate the output from the subprocess or ignore it

My problem is, that sometimes I'd like to ignore the output. For example, I'd like to send 1M commands to the subprocess, ignore the result (to speed up the simulation time) and check only the output of the last one.

However, it seems (this is my suspicion only!), that I have to consume the stdout buffer. Otherwise, it hangs forever...

Here is an example:

from subprocess import Popen, PIPE

class Simulation:
    def __init__(self, path):
        self.proc = Popen([path], stdin=PIPE, stdout=PIPE)

    def executeCmd(self, cmd):
        self.proc.stdin.write(cmd.encode('utf-8'))
        self.proc.stdin.flush()
        output = ""
        line = ""

        while '</end>' not in line:
            line = self.proc.stdout.readline().decode('utf-8')
            output += line

        return output
        
    def executeCmd_IgnoreOutput(self, cmd):
        self.proc.stdin.write(cmd.encode('utf-8'))
        self.proc.stdin.flush()
        ## self.proc.stdout.read() #< can't do that since subprocess is still running and there is no EOF sign
        self.proc.stdout.flush() #< does not clear the buffer :(

    def tearDown(self):
        self.proc.stdin.write("exit_command")
        self.proc.stdin.flush()
        exit_code = self.proc.wait()
    
simulation = Simulation("\path\to\binary")
output = simulation.executeCmd("command")
#do something with the output

for i in range(1000000):
    simulation.executeCmd_IgnoreOutput("command") #hangs after few thousand iterations
simulation.tearDown()
    

executeCmd consumes the whole output (I cannot use read since the subprocess is still running and there is no EOF at the end of the output`). But this is very expensive - I have to iterate through all lines...

So my prototype was to create executeCmd_IgnoreOutput which doesn't consume the buffer. But it hangs after a few thousand iterations.

My questions are:

• Maybe I made a mistake at the very beginning- is the subprocess package suitable for usage as above? Maybe I should use a different tool for such a purpose...
• If so, then how can I clear up the stdout buffer? (flush doesn't work in that case - it still hangs)
• Or maybe it hangs for different reasons (any ideas?)

Answer 1

To drain the stdout pipe, make sure it isn't buffered, then read it in chunks until there's nothing to read:

import subprocess


class Simulation:
    def __init__(self, path):
        self.proc = subprocess.Popen(
            [path],
            stdin=subprocess.PIPE, 
            stdout=subprocess.PIPE,
            bufsize=0,  # unbuffered
        )

    def executeCmd(self, cmd):
        self.proc.stdin.write(cmd.encode("utf-8"))
        self.proc.stdin.flush()
        output = b""
        while True:
            data = self.proc.stdout.read(65536)
            if not data:
                break
            output += data
        return output

    def tearDown(self):
        self.executeCmd("exit_command")
        return self.proc.wait()