Subprocess popen: Why do all writes happen at once in child process?

Question

I have two scripts, one controlling the other and communicating to it via stdin. The parent script:

import subprocess
import time

p = subprocess.Popen(['python','read_from_stdin.py'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)

for i in range(0,10):
    p.stdin.write(str(i))
    p.stdin.write('\r\n') # \n is not sufficient on Windows
    p.stdin.flush()
    print i
    time.sleep(1)

p.stdin.close()

The child script (called 'read_from_stdin.py'):

import sys
import datetime

with open(datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S') + '.txt','w') as f:

    for line in sys.stdin:
        f.write(datetime.datetime.now().isoformat() + ' ' + line)

In the file that's created by the child script all the inputs have the same timestamp, despite being written a second apart by the parent script, and despite the use for flush().

score 1 · Answer 1 · edited May 23 '17 at 12:15

1

EDIT: As per Karoly Horvath's comment below, it's not the case that it waits for for EOF, but there's buffering. The different child script below does work as expected.

I found this question on the subject: How do you read from stdin in Python?

A fair way down in the answers is:

The answer proposed by others:
for line in sys.stdin:
  print line
is very simple and pythonic, but it must be noted that the script will wait until EOF before starting to iterate on the lines of input.

This child script behaves as expected:

import sys
import datetime

with open(datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S') + '.txt','w') as f:

    line = sys.stdin.readline()
    while line:
        f.write(datetime.datetime.now().isoformat() + ' ' + line)
        line = sys.stdin.readline()

    f.write('Finished')

edited May 23 '17 at 12:15

Community

1
1

answered Jan 22 '16 at 23:19

user1379351

723
1
5
18

1

Just wanted to post :) http://stackoverflow.com/questions/8416586/turn-off-buffering It *doesn't* wait for EOF, but there's *buffering*. Please update your answer. – Karoly Horvath Jan 22 '16 at 23:24
@KarolyHorvath: it is correct that it doesn't wait for EOF but the question you've linked is unrelated (the fix `grep --line-buffered` won't work here -- OP already calls `p.stdin.flush()`). The issue is the read-ahead bug in Python 2 with `for line in sys.stdin`. The workaround is: `for line in iter(sys.stdin.readline, '')` – jfs Jan 23 '16 at 07:51
@KarolyHorvath : I saw it. The solution works but there is no indication in the answer about the actual cause (bug in CPython 2) – jfs Jan 23 '16 at 11:01
You saw it now. And yes, it's a bug. – Karoly Horvath Jan 23 '16 at 11:13
@KarolyHorvath: If you are suggesting that I haven't seen it before your comment or haven't understood it then you are wrong. Look at [my answers in the `subprocess` tag](http://stackoverflow.com/search?q=user%3A4279+%5Bsubprocess%5D). If you want me to be notified about your comments then use @ username syntax. – jfs Jan 23 '16 at 20:43

score 1 · Accepted Answer · answered Jan 23 '16 at 07:53

1

It is the read-ahead bug in Python 2: for line in sys.stdin: doesn't yield anything until its internal buffer is full. Use for line in iter(sys.stdin.readline, ''):, to workaround it.

answered Jan 23 '16 at 07:53

jfs

399,953
195
994
1,670

Subprocess popen: Why do all writes happen at once in child process?

2 Answers2