15

I'm trying to talk to a child process using the python subprocess.Popen() call. In my real code, I'm implementing a type of IPC, so I want to write some data, read the response, write some more data, read the response, and so on. Because of this, I cannot use Popen.communicate(), which otherwise works well for the simple case.

This code shows my problem. It never even gets the first response, hangs at the first "Reading result". Why? How can I make this work as I expect?

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
print "Reading result:"
print p.stdout.readline()

p.stdin.write("cat\n")
print "Reading result:"
print p.stdout.readline()
Mats Ekberg
  • 1,695
  • 4
  • 15
  • 23

2 Answers2

7

sed's output is buffered and only outputs its data until enough has been cumulated or the input stream is exhausted and closed.

Try this:

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
p.stdin.write("cat\n")
p.stdin.close()

print "Reading result 1:"
print p.stdout.readline()

print "Reading result 2:"
print p.stdout.readline()

Be aware that this cannot be done reliably which huge data as wriring to stdin blocks once the buffer is full. The best way to do is using communicate().

glglgl
  • 89,107
  • 13
  • 149
  • 217
  • But sed does not act that way when running the command in the shell directly. If you do that, the response comes after each line. And the problem remains even if I call p.stdin.flush() after writing. Also, in real life, I have written the called program as well, there is no buffering and it behaves in the same way. I'm not convinced that buffering is the problem here. – Mats Ekberg Apr 28 '12 at 15:19
  • add bufsize=0 can solve your problem. `subprocess.Popen(['sed', '-l', 's/a/x/g'], bufsize=0, stdin=PIPE, stdout=PIPE)` – codeskyblue Nov 27 '19 at 02:34
5

I would try to use Popen().communicate() if you can as it does a lot of nice things for you, but if you need to use Popen() exactly as you described, you'll need to set sed to flush its buffer after newlines with the -l option:

p = subprocess.Popen(['sed', '-l', 's/a/x/g'],
                     stdout=subprocess.PIPE,
                     stdin=subprocess.PIPE)

and your code should work fine

gnr
  • 2,324
  • 1
  • 22
  • 24
  • 1
    Indeed it does work! My sed for some reason uses -u for "unbuffered", not -l, but it works all the same. This solves my example code, but unfortunately not my real code, as the actual command is not sed, but another python program. Good reply though, you pinpointed the problem. – Mats Ekberg Apr 28 '12 at 15:39
  • 1
    Yup, problem solved. The problem was output buffering of the result. Doing a simple stdout.flush() in my subprocess solved the problem. Thanks! – Mats Ekberg Apr 28 '12 at 15:51