How to make subprocess only communicate error

Question

We have created a commodity function used in many projects which uses subprocess to start a command. This function is as follows:

def _popen( command_list ):
    p = subprocess.Popen( command_list, stdout=subprocess.PIPE,
        stderr=subprocess.PIPE )

    out, error_msg = p.communicate()

    # Some processes (e.g. system_start) print a number of dots in stderr
    # even when no error occurs.
    if error_msg.strip('.') == '':
        error_msg = ''

    return out, error_msg

For most processes this works as intended.

But now I have to use it with a background-process which need to keep running as long as my python-script is running as well and thus now the fun starts ;-).
Note: the script also needs to start other non background-processes using this same _popen-function.

I know that by skipping p.communicate I can make the process start in the background, while my python script continues.
But there are 2 problems with this:

1. I need to check that the background process started correctly
2. While the main process is running I need to check the stdout and stderr of the background process from time to time without stopping the process / ending hanging in the background process.

Check background process started correctly
For 1 I currently adapted the _popen version to take an extra parameter 'skip_com' (default False) to skip the p.communicate call. And in that case I return the p-object i.s.o. out and error_msg. This so I can check if the process is running directly after starting it up and if not call communicate on the p-object to check what the error_msg is.

MY_COMMAND_LIST = [ "<command that should go to background>" ]

def _popen( command_list, skip_com=False ):    
    p = subprocess.Popen( command_list, stdout=subprocess.PIPE,
        stderr=subprocess.PIPE )

    if not skip_com:
        out, error_msg = p.communicate()

        # Some processes (e.g. system_start) print a number of dots in stderr
        # even when no error occurs.
        if error_msg.strip('.') == '':
            error_msg = ''

        return out, error_msg
    else:
        return p

...
p = _popen( MY_COMMAND_LIST, True )
error = _get_command_pid( MY_COMMAND_LIST ) # checks if background command is running using _popen and ps -ef
if error:
    _, error_msg = p.communicate()

I do not know if there is a better way to do this.

check stdout / stderr
For 2 I have not found a solution which does not cause the script to wait for the end of the background process.
The only ways I know to communicate is using iter on e.g. p.stdout.readline. But that will hang if the process is still running:

for line in iter( p.stdout.readline, "" ): print line

Any one an idea how to do this?

/edit/ I need to check the data I get from stdout and stderr seperately. Especially stderr is important in this case since if the background process encounters an error it will exit and I need to catch that in my main program to be able to prevent errors caused by that exit.

The stdout output is needed in some situations to check the expected behaviour in the background process and to react on that.

Answer 1

Update

The subprocess will actually exit if it encounters an error

If you don't need to read the output to detect an error then redirect it to DEVNULL and call .poll() to check child process' status from time to time without stopping the process.

---

^{assuming you have to read the output:}

Do not use stdout=PIPE, stderr=PIPE unless you read from the pipes. Otherwise, the child process may hang as soon as any of the corresponding OS pipe buffers fill up.

If you want to start a process and do something else while it is running then you need a non-blocking way to read its output. A simple portable way is to use a thread:

def process_output(process):
    with finishing(process): # close pipes, call .wait()
        for line in iter(process.stdout.readline, b''):
            if detected_error(line):
                communicate_error(process, line) 


process = Popen(command, stdout=PIPE, stderr=STDOUT, bufsize=1)
Thread(target=process_output, args=[process]).start()

I need to check the data I get from stdout and stderr seperately.

Use two threads:

def read_stdout(process):
    with waiting(process), process.stdout: # close pipe, call .wait()
        for line in iter(process.stdout.readline, b''):
            do_something_with_stdout(line)

def read_stderr(process):
    with process.stderr:
        for line in iter(process.stderr.readline, b''):
            if detected_error(line):
                communicate_error(process, line) 

process = Popen(command, stdout=PIPE, stderr=PIPE, bufsize=1)
Thread(target=read_stdout, args=[process]).start()
Thread(target=read_stderr, args=[process]).start()

You could put the code into a custom class (to group do_something_with_stdout(), detected_error(), communicate_error() methods).

Answer 2

It may be better or worse than what you imagine...

Anyway, the correct way of reading a pipe line by line is simply:

for line in p.stdout:
    #process line is you want of just
    print line

Or if you need to process that inside of a higher level loop

line = next(p.stdout)

But a harder problem could come from the commands started from Python. Many programs use the underlying C standard library, and by default stdout is a buffered stream. The system detects whether the standard output is connected to a terminal, and automatically flushes output on a new line (\n) or on a read on same terminal. But if output is connected to a pipe or a file, everything is buffered until the buffer is full, which on current systems requires several kBytes. In that case nothing can be done at Python level. Above code would get a full line as soon as it would written on the pipe, but cannot guess before callee has actually written something...