Pipe unbuffered stdout from subprocess to websocket

Question

How would you pipe the stdout from subprocess to the websocket without needing to wait for a newline character? Currently, the code below only sends the stdout on a newline.

Code attached for the script being run by the subprocess. Is the output not being flushed properly from there?

send_data.py:

import asyncio
import websockets
import subprocess
import sys
import os

async def foo(websocket, path):
        print ("socket open")
        await websocket.send("successfully connected")

        with subprocess.Popen(['sudo','python3', '-u','inline_print.py'],stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=0, universal_newlines=True) as p:
                for line in p.stdout:
                    line = str(line.rstrip())
                    await websocket.send(line)
                    p.stdout.flush()
                for line in p.stderr:
                    line = str(line.rstrip())
                    await websocket.send(line)
                    p.stdout.flush()


start_server = websockets.serve(foo, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

inline_print.py:

from time import sleep
import sys

loading = 'LOADING...LOADING...LOADING...LOADING...LOADING...'
for i in range(50):
    print(loading[i], sep='', end=' ', flush=True)
    sleep(0.1)

if end=' ' is changed to end='\n' then the stdout from send_data.py occurs in realtime.

js client:

var ws = new WebSocket('ws://localhost:8765/');

ws.onmessage = function(event) {
  console.log(event.data);
};

I acknowledge this question is similar to these:

catching-stdout-in-realtime-from-subprocess

how-do-i-get-real-time-information-back-from-a-subprocess-popen-in-python-2-5

intercepting-stdout-of-a-subprocess-while-it-is-running

yet none of the solutions work without a newline character from the subprocess.

gelonida · Accepted Answer · 2020-05-04T12:33:29.510

If you write

      for line in p.stdout:

then you (kind of) implicitly say, that you want to wait for a complete line

you had to use read(num_bytes) and not readline()

Below one example to illustrate:

sub.py: (example subprocess)

import sys, time
for v in range(20):
    print(".", end="")
    sys.stdout.flush()
    if v % 4 == 0:
        print()
    if v % 3 != 0:
        time.sleep(0.5)

rdunbuf.py: (example reading stddout unbuffered)

contextlib, time, subprocess

def unbuffered(proc, stream='stdout'):
    stream = getattr(proc, stream)
    with contextlib.closing(stream):
        while True:
            last = stream.read(80) # read up to 80 chars
            # stop when end of stream reached
            if not last:
                if proc.poll() is not None:
                    break
            else:
                yield last

# open subprocess without buffering and without universal_newlines=True
proc = subprocess.Popen(["./sub.py"], stdout=subprocess.PIPE, bufsize=0)

for l in unbuffered(proc):
    print(l)
print("end")

Please note as well, that your code might block if it produces a lot of error messages before producing normal output, as you try first to read all normal output and only then data from stderr.

You should read whataver data your subprocess produces as before any pipeline buffers are blocking independently whether this is stdout or stderr. You can use select.select() ( https://docs.python.org/3.8/library/select.html#select.select ) in order to decide whether you had to read from stdout, or stderr

I noticed I also needed to remove `universal_newlines=True` – siddhant1999 May 04 '20 at 02:25 — siddhant1999, May 04 '20 at 02:25

Pipe unbuffered stdout from subprocess to websocket

1 Answers1