subprocess output to stdout and to PIPE

Question

I am using the subprocess module like that:

ping = subprocess.Popen('fping.exe 192.168.2.3 196.65.58.69', stdout=PIPE)
output = ping.stdout.readlines()

I am need the output list in order to process it later at the program, but it seems since the stdout is directed to PIPE it isn't output the results to the console. I would like to get both the console output (as it being executed) and the output list.

How can i do that ?

I have done a search, and got an answer here, but i am unable to implement it.

I am using Python 3.x on Windows env.

Thanks.

You can always echo the output in your script: `print(output)` — Lev Levitsky, Mar 25 '12 at 10:36
@LevLevitsky I would like to get the console output as the command being executed. — Hanan, Mar 25 '12 at 10:42
What about [`os.dup2`](http://docs.python.org/release/3.0.1/library/os.html#file-descriptor-operations) then? — Lev Levitsky, Mar 25 '12 at 10:49

osexp2000 · Answer 1 · 2023-07-11T12:42:14.107

Following solution works, a bit cumbersome, but very convenient.

It uses Bash's command substitution, essentially similar to create 2 threads to consume stdout and stderr and tee them to parent's stdout and stderr.

Please replace the var cmd with your own command, it could be any command, then try run following script (just copy all and paste to run), then wait for 5 seconds, it will automatically stop.

python3 <<'PYTHON_SCRIPT_EOF'

# change this command to yours. It may accept arguments such as $1, $2, then please pass arguments after wrapper_cmd, such as '--', 'arg1', 'arg2'
cmd = '''
  echo test arg1 is "$1"
  while (( ++ i <= 5 )); do
    date;
    echo for stderr >&2
    sleep 1
  done
'''


import subprocess, os
if not os.environ.get("PARENT_STDERR"):
  parent_stderr = os.dup(2)
  os.set_inheritable(parent_stderr, True)
  os.environ["PARENT_STDERR"] = str(parent_stderr)


# the magic 2> >(tee ...) and > >(tee ...) command can make the command output stdout/stderr to parent stderr REAL TIME, yet can capture the stdout/stderr.
# See https://tldp.org/LDP/abs/html/process-sub.html
# I could replace the `> >(tee ...)` with `| tee ...` but that affect the exit code of the whole command, require set -e -o pipefail;
wrapper_cmd = '(\n' + cmd + '\n) 2> >(tee /proc/self/fd/$PARENT_STDERR >&2) > >(tee /proc/self/fd/$PARENT_STDERR)'

p = subprocess.Popen([ 'bash', '-c', wrapper_cmd, '--', 'arg1_test_value' ],
          stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, close_fds=False)
stdout, stderr = p.communicate()


print("====captured stdout:\n", stdout)
print("====captured stderr:\n", stderr)

PYTHON_SCRIPT_EOF

The output:

test arg1 is arg1_test_value

continue: you can see it output data belongs to stdout and stderr to current terminal real time:

Tue Jul 11 12:21:13 UTC 2023
for stderr
Tue Jul 11 12:21:14 UTC 2023
for stderr
Tue Jul 11 12:21:15 UTC 2023
for stderr
Tue Jul 11 12:21:16 UTC 2023
for stderr
Tue Jul 11 12:21:17 UTC 2023
for stderr

yet can get stdout and stderr respectively:

====captured stdout:
 test arg1 is arg1_test_value
Tue Jul 11 12:21:13 UTC 2023
Tue Jul 11 12:21:14 UTC 2023
Tue Jul 11 12:21:15 UTC 2023
Tue Jul 11 12:21:16 UTC 2023
Tue Jul 11 12:21:17 UTC 2023

====captured stderr:
 for stderr
for stderr
for stderr
for stderr
for stderr

score 0 · Answer 2 · answered Mar 26 '12 at 04:00

There's no such thing as a pipe that goes to two places. Everything written to a pipe will only be read once. (While it's theoretically possible for your program and the console to have access to the same out pipe, if you succeed in doing so then only some of the data will go to your program, and only the data that doesn't will end up on the console.) To get all the output to your program and to the console, someone will have to read and duplicate the data. On a unix-like system, you might use the "tee" command for this, but you probably don't have that on your Windows machine.

So you will have to write the output to the console as you get it.

In this case, you can probably get away with using readline() in a loop instead of readlines().

score 0 · Accepted Answer · answered Apr 03 '12 at 06:51

I have found a way to do that here is it:

for line in os.popen("Fping x.x.x.x x.x.x.x -l"):
    ipList.append(line)
    print(line)

That way, i am able to get the results from the Fping program into the list, and to print it to the screen while it is executing, since the for loop with the os.popen aren't wait for the program to finish, but always looping at every line from the program.

subprocess output to stdout and to PIPE

3 Answers3

Linked