I have been trying to troubleshoot subprocess.PIPE with subprocesses with no luck.
I'm trying to pass commands to an always running process and receive the results without having to close/open the process each time.
Here is the main launching code:
launcher.py:
import subprocess
import time
command = ['python', 'listener.py']
process = subprocess.Popen(
command, bufsize=0,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
# simulates sending a new command every 10 seconds
for x in range(1,10):
process.stdin.write(b'print\r\n')
process.stdin.flush()
time.sleep(10)
listener.py:
import sys
file = open('log.txt', 'w+')
while True:
file.write(sys.stdin.read(1))
file.close()
This is simplified to show relevent pieces. In the end I'll have threads listening on the stdout and stderr but for now I'm trying to troubleshoot the basics.
What I expect to happen: for each loop in launcher.py, the file.write() in listener.py would write. What happens instead: everything writes when the loop closes and the program terminates, or I SIGTERM / CTRL-C the script.
I'm running this in Windows 8 Python 3.4.
It's almost as if stdin buffers until the process closes and then it passes through. I have buffsize=0 set, and I'm flushing, so that doesn't make sense to me. I thought either one or the other would be sufficient.
The subprocess is running in a different process, so the sleep in launcher should have no impact on the subprocess.
Does anyone have any ideas why this is blocking?
Update 1: The same behaviour is also seen with the following code run from the console (python.exe stdinreader.py)
That is, when you type into the console while the program is running, nothing is written to the file.
stdinreader.py:
import sys
import os
file = open('log.txt', 'w+b')
while True:
file.write(sys.stdin.read(1))
file.close()
Adding a file.flush() just before file.write() solves this, but that doesn't help me with the subprocess because I don't have control of how subprocess flushes (which would be my return subprocess.PIPE). Maybe if I reinitialize that PIPE with open('wb') it will not buffer. I will try.
Update 2: I seem to have isolated this problem to the subprocess being called which is not flushing after it's writes to stdout.
Is there anything I can do to force a flush on the stdout PIPE between parent and child without modifying the subprocess? The subprocess is magick.exe (imagemagick 7) running with args ['-script, '-']. From the point of view of the subprocess it has a stdout object of <_io.TextIOWrapper name='' mode='w' encoding='cp1252'>. I guess the subprocess will just open the default stdout objects on initialization and we can't really control whether it buffers or not.
The strange thing is that passing the child the normal sys.stdout object instead of subprocess.PIPE does not require the subprocess to .flush() after write.
