Live stream stdout and stdin with websocket

Question

online compiler this is my website where users can run console programs.

At present, the user has to enter the program input before running the program. I am trying to build live user input for the program(want to give the same experience as they run programs on their laptop).

In research to achieve this I came across a solution to stream stdout and stdin with websocket.

My implementation

# coding: utf-8
import subprocess
import thread

from tornado.websocket import WebSocketHandler

from nbstreamreader import NonBlockingStreamReader as NBSR


class WSHandler(WebSocketHandler):
    def open(self):
        self.write_message("connected")
        self.app = subprocess.Popen(['sh', 'app/shell.sh'], stdout=subprocess.PIPE, stdin=subprocess.PIPE,
                                    shell=False)
        self.nbsr = NBSR(self.app.stdout)
        thread.start_new_thread(self.soutput, ())

    def on_message(self, incoming):
        self.app.stdin.write(incoming)

    def on_close(self):
        self.write_message("disconnected")

    def soutput(self):
        while True:
            output = self.nbsr.readline(0.1)
            # 0.1 secs to let the shell output the result
            if not output:
                print 'No more data'
                break
            self.write_message(output)

nbstreamreader.py

from threading import Thread
from Queue import Queue, Empty


class NonBlockingStreamReader:
    def __init__(self, stream):
        '''
        stream: the stream to read from.
                Usually a process' stdout or stderr.
        '''

        self._s = stream
        self._q = Queue()

        def _populateQueue(stream, queue):
            '''
            Collect lines from 'stream' and put them in 'quque'.
            '''

            while True:
                line = stream.readline()
                if line:
                    queue.put(line)
                else:
                    raise UnexpectedEndOfStream

        self._t = Thread(target=_populateQueue,
                         args=(self._s, self._q))
        self._t.daemon = True
        self._t.start()  # start collecting lines from the stream

    def readline(self, timeout=None):
        try:
            return self._q.get(block=timeout is not None,
                               timeout=timeout)
        except Empty:
            return None


class UnexpectedEndOfStream(Exception): pass

shell.sh

#!/usr/bin/env bash
echo "hello world"
echo "hello world"
read -p "Your first name: " fname
read -p "Your last name: " lname
echo "Hello $fname $lname ! I am learning how to create shell scripts"

This code streams stdout un-till shell.sh code reaches read statement.

Please guide me what wrong I am doing. Why it doesn't wait for stdin and reaches print 'No more data' before complete program executions?

Source code to test it https://github.com/mryogesh/streamconsole.git

Yes,. It's going inside on_message method and after thread.start_new_thread(self.soutput, ()) this statement i added print statement which means thread is not blocked. — Yogesh Lakhotia, Jan 03 '17 at 14:01
What happend when you (after your input) hit enter on the "first name" statement? Is it simply not streamed? What happend if you go through your whole `shell.sh`, even the last `echo` isn't streamed? The issue reminds me of something (I did around a year ago exactly this, syso streaming through websocket, but currently I don't have access to that code; but I remember having problems with premature termination of the streams too). — D. Kovács, Jan 05 '17 at 15:09
Stream terminate as soon as it reaches read statement. output is "hello world\nhello world" — Yogesh Lakhotia, Jan 05 '17 at 16:14
I have uploaded code on github https://github.com/mryogesh/streamconsole.git . In case you need to test code. — Yogesh Lakhotia, Jan 05 '17 at 16:16

score 5 · Accepted Answer · edited May 23 '17 at 12:01

Your readline() method times out unless you send input within 100ms, which then breaks the loop. The reason you don't see the read -p prompt is buffering (because of readline and pipe buffering). Finally, your example javascript doesn't send a trailing newline, so read will not return.

If you increase the timeout, include a newline, and find a way to work around buffering issues, your example should basically work.

I'd also use tornado.process and coroutines instead of subprocess and thread:

from tornado import gen
from tornado.process import Subprocess
from tornado.ioloop import IOLoop
from tornado.iostream import StreamClosedError
from tornado.websocket import WebSocketHandler


class WSHandler(WebSocketHandler):
    def open(self):
        self.app = Subprocess(['script', '-q', 'sh', 'app/shell.sh'], stdout=Subprocess.STREAM, stdin=Subprocess.STREAM)
        IOLoop.current().spawn_callback(self.stream_output)

    def on_message(self, incoming):
        self.app.stdin.write(incoming.encode('utf-8'))

    @gen.coroutine
    def stream_output(self):
        try:
            while True:
                line = yield self.app.stdout.read_bytes(1000, partial=True)
                self.write_message(line.decode('utf-8'))
        except StreamClosedError:
            pass

well explained. Thanks for answering :) – Yogesh Lakhotia Jan 07 '17 at 22:33 — Yogesh Lakhotia, Jan 07 '17 at 22:33

Live stream stdout and stdin with websocket

1 Answers1

Linked