Read stream and pass to subprocess

Question

I'm writing a python script that reads a stream from stdin and passes this stream to subprocess for further processing. The problem is that python hangs after having processed the input stream.

For example, this toy program sorter.py should read from stdin and pass the stream to subprocess for sorting via Unix sort:

cat dat.txt | ./sorter.py

Here's sorter.py:

#!/usr/bin/env python

import subprocess
import sys

p= subprocess.Popen('sort -', stdin= subprocess.PIPE, shell= True)
for line in sys.stdin:
    p.stdin.write(line)
sys.exit()

The stream from cat is correctly sorted but the programs hangs, i.e. sys.exit() is never reached.

I've read quite a few variations on this theme but I can't get it right. Any idea what is missing?

Thank you!

Dario

Answer 1

My guess: sys.exit() is reached but sort continues to run. You should close p.stdin pipe to signal EOF to sort:

#!/usr/bin/env python2
import subprocess
import sys

p = subprocess.Popen('sort', stdin=subprocess.PIPE, bufsize=-1)
with p.stdin:
    for line in sys.stdin:
        # use line here
        p.stdin.write(line)
if p.wait() != 0:
    raise Error

Example:

$ < dat.txt ./sorter.py

If you don't need to modify the stdin stream then you don't need to use PIPE here:

#!/usr/bin/env python
import subprocess

subprocess.check_call('sort')

Answer 2

you probably have a problem with buffering -the OS doesnt send the data the moment it arrives to stdin - check this out

https://www.turnkeylinux.org/blog/unix-buffering