Writing to a redirected input of spawned process in Python

Question

Sorry, I am really not a python person, but the situation hit me, and this will be something very obvious for you python people.

I am trying to communicate interactively with a spawned console process (in our use case it is a console program that communicates with an older, but still nice HP instrument for measuring some physics variables in an experiment).
I have found some insightful example here:
Running interactive program from within python

But when trying to get inspired (I do not want that timer), I started to write this from scratch, testing with an ftp.exe program that is typically present on a windows box, with the following:

#!/usr/bin/python
import subprocess, sys
mystdout=sys.stdout
p = subprocess.Popen("ftp.exe", stdin=subprocess.PIPE, stdout=subprocess.PIPE) #, shell=True)
sys.stdout=p.stdin
print('bye\n'.encode())
sys.stdout=mystdout
print('done')

But the print('bye\n'.encode()) results in:

print('bye\n'.encode())
TypeError: a bytes-like object is required, not 'str'

I can't use the communicate() method of subprocess as it seems to be not very interactive (one-time only).

Would you, please, give me a hint where is my dumb? This machine runs Python 3.6.1 on Windows, but a friendlier Linux machine with 3.7.3 gives the same greeting.

Palo · Accepted Answer · 2020-11-11T12:59:31.980

Interesting, I just figured out that adding

universal_newlines=True

as the last argument of Popen() solves my problem. I guess the stream did not like the print() because it was open in binary mode. Adding this flag opens the stream in text mode. I will post a complete example including interactive communication when I have it.

Edit:

Oh, now I found some more resources that are relevant to this issue.

It seems it is easier on Linux as there is the fcntl package that allows setup the pipes with non-blocking read(). Unfortunately, it seems it is not available on Windows :( I was hoping to use the peek() call, but surprise, surprise, it is also blocking on pipes! :(

So we are stuck either with threads or perhaps with the modern hard-to-read async/await style.

Based on the inspiration linked above, I have reached the following simple working ftp example:

#!/usr/bin/python

import time, sys, subprocess, threading, queue

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()    

def getOutput(outQueue):
    outStr = ''
    try:
        while True: 
            outStr+=outQueue.get_nowait()

    except queue.Empty:
        return outStr           

p = subprocess.Popen("ftp.exe", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) 

outQueue = queue.Queue()
errQueue = queue.Queue()

outThread = threading.Thread(target=enqueue_output, args=(p.stdout, outQueue))
errThread = threading.Thread(target=enqueue_output, args=(p.stderr, errQueue))

outThread.daemon = True
errThread.daemon = True

outThread.start()
errThread.start()

p.stdin.write('status\n')
p.stdin.flush()
time.sleep(0.2)

errors = getOutput(errQueue)
output = getOutput(outQueue)

print("err:" + errors)
print("out:" + output)

time.sleep(2)

p.stdin.write('help\n')
p.stdin.flush()
time.sleep(0.2)

errors = getOutput(errQueue)
output = getOutput(outQueue)

print("err:" + errors)
print("out:" + output)

time.sleep(2)

p.stdin.write('bye\n')
p.stdin.flush()
time.sleep(0.2)

errors = getOutput(errQueue)
output = getOutput(outQueue)

print("err:" + errors)
print("out:" + output)

time.sleep(2)

print('done')

One last note: when using this to communicate with my own c++ program, I had to issue fflush(stdout) after each output. Not even a newline did help the stuff to fall into the pipe without flushing.

Writing to a redirected input of spawned process in Python

1 Answers1