Stream output from server-side .exe through PHP into local Python

Question

I have a situation where I am running a Python script locally and using f = urllib.urlopen() to query a server and pass some parameters. A PHP script on the server side accepts the parameters, checks them, and passes them to an executable using the passthru() command. The executable runs, and then the output is passed back to the client, where it is read using response = f.read(). The response is then printed.

This works, however the output is returned in one big block at the end of the process. What I would like to do is have the output from the online executable print in the python terminal locally as it runs. Does anyone know a good approach to take here? I've tried for line in f: print line, but that gives the same behavior as before.

Have you tried flushing? http://stackoverflow.com/questions/230751/how-to-flush-output-of-python-print — Moses Koledoye, Jan 30 '17 at 21:58
That sounds like the PHP script dumps the data in one big block into the output at once, wouldn't it be better to change the PHP script? — Alex, Jan 30 '17 at 22:02
Does the server stream the command output when you access the url via a web browser? If the PHP script collects all the output before sending it out, then you'll need to fix the PHP script. But if it already streams the data, then you could use my answer below to process the streamed data as it comes in. — Matthias Fripp, Jan 30 '17 at 22:24

Matthias Fripp · Accepted Answer · 2017-01-30T22:37:54.027

The documentation for urllib says that "the read() method, if the size argument is omitted or negative, may not read until the end of the data stream; there is no good way to determine that the entire stream from a socket has been read in the general case."

So you might try processing one character at a time, like this:

line = ''
while True:
    char = f.read(1)
    if not char:
        break
    elif char == '\n':
        print line
        line = ''
    else:
        line += char

# or a tidier form based on
# http://bob.ippoli.to/archives/2005/06/14/python-iterators-and-sentinel-values/
line = ''
for char in iter(lambda: f.read(1), ''):
    if char == '\n':
        print line
        line = ''
    else:
        line += char

I ended up having to change both the Python and PHP scripts, but this worked on the Python side. For the PHP script, I changed over to using popen instead of passthrough, and had to enforce no buffering: http://stackoverflow.com/questions/20107147/php-reading-shell-exec-live-output — aknight0, Jan 31 '17 at 19:09

Stream output from server-side .exe through PHP into local Python

1 Answers1