How to stream messages in a pipe from one process to another?

Question

I have 2 python (2.7) processes.
The parent process needs to send rows of text to a child process, and the child process should process them as they come in (not wait for the parent process to finish).

I have this code which doesn't work:

# Sender
import subprocess

process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)

try:
    while True:
        process.stdin.write(msg + '\n')  # 'msg' is a changing string
        # process.stdin.flush() <-- commented out since it doesn't help
except KeyboardInterrupt:
    process.stdin.close()
    process.wait()

And the child process:

# Receiver
import sys

for line in sys.stdin:
    print line.strip()

The problem is that the child process waits until the parent process exits before it prints out the messages.

What I'm trying to achieve is a child process that processes the messages as soon as they are written to the pipe.

Patrik Iselind · Answer 1 · 2018-02-11T13:28:42.933

Try adding a process.stdin.flush() after your process.stdin.write(). That way you actually send the string to the other process. What you're suffering from here is your kernel caching everything you write. It does this to be more efficient when actually sending the data to the other process. flush force the kernel to send your data regardless of how full the kernel's buffer is.

I tried your code as such:

# Sender
import subprocess                                                                                                                                                                                           

process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
msg = "This is my message"

try:
    while True:
        process.stdin.write(msg + '\n')  # 'msg' is a changing string
        process.stdin.flush() # This code works well for me regardless of the presence of this line
except KeyboardInterrupt:
    process.stdin.close()
    process.wait()

# Receiver
import sys

for line in sys.stdin:
    print line.strip()

With "works well" here i mean that i get "This is my message" printed as fast as the computer can perform. I'm trying this in Python 2.7.12 for the record.

tried it, but the result is the same. I'll edit my question to add the flush command — user1102018, Feb 11 '18 at 13:02
it's a string. I simplified my "sender" function. you can replace it with a hardcoded string. — user1102018, Feb 11 '18 at 13:11
Could you please add in your sender function as well? Perhaps that's where the problem is? — Patrik Iselind, Feb 11 '18 at 18:29

score 0 · Answer 2 · answered Feb 11 '18 at 14:07

The story of how buffering works for sys.stdin and sys.stdout has made me cry more than once. A similar problem is discussed in Setting smaller buffer size for sys.stdin?.

As to your specific problem, I suggest you change your child to use sys.stdin.readline() instead of iterating over sys.stdin. The former somewhat "buffers less" :)

while True:
    line = sys.stdin.readline()
    if not line: break
    print (line.strip())

In the parent, you'll likely either need to set bufsize=0 in your call to Popen (making your pipe completely unbuffered), or you'll need the process.stdin.flush() line, as Patrik suggests. I'd opt for the latter.

Tested on Python 2.7.14 on Windows 10 64bit.

How to stream messages in a pipe from one process to another?

2 Answers2