How to send the output of a long running Python script over a WebSocket?

Question

A user uploads a python file that I need to execute on my server and send back the stdout that's created over a WebSocket. The python file that's executed will run for several minutes and I need to return the stdout over a socket as they are "printed" out in real-time, not at the completion of the script.

I've tried using: Python. Redirect stdout to a socket, but that's not a WebSocket and my React frontend can't connect to it successfully. (if you can solve that, that would also solve my problem)

I've also tried using websocketd but since I can't add sys.stdout.flush() after each of the users' added print statements it doesn't solve my problem.

I've also tried using subprocess's PIPE functionality but that has the same flush issue

async def time(websocket, path):
    while True:
        data = "test"
        await websocket.send(data)
        # Run subprocess to execute python file in here
        # sys.stdout => websocket.send             

start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

This is the python test script I am using:

from time import sleep
for i in range(40):
    print(i)
    sleep(0.1)

cant you create a file-like object, set `sys.stdout` to it, and override the`write` function to send the data? — בנימין כהן, Aug 18 '19 at 03:11
@בנימיןכהן How would I go about doing that? That does sound like a solution — siddhant1999, Aug 18 '19 at 03:42

score 5 · Accepted Answer · 2019-08-19T05:07:47.287

This stand-alone example will

Read a python script from a web socket
Write the script to the file system
Run the script with output buffering disabled
Read the script output one line at a time
Write each line of output to the web socket

import asyncio
import websockets
import subprocess

async def time(websocket, path):
    script_name = 'script.py'
    script = await websocket.recv()
    with open(script_name, 'w') as script_file:
        script_file.write(script)
    with subprocess.Popen(['python3', '-u', script_name],
                          stdout=subprocess.PIPE,
                          bufsize=1,
                          universal_newlines=True) as process:
        for line in process.stdout:
            line = line.rstrip()
            print(f"line = {line}")
            await websocket.send(line)

start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

I used this javascript code to test the server:

const WebSocket = require('ws');
let socket = new WebSocket("ws://127.0.0.1:5678");

socket.onopen = function(e) {
    let script = '\
import time\n\
for x in range(100):\n\
    print(f"x = {x}")\n\
    time.sleep(0.25)\n\
';
    console.log("sending data...");
    socket.send(script);
    console.log("done.");
};

socket.onmessage = function(event) {
    console.log(event.data.toString());
};

socket.onerror = function(event) {
    console.log(event);
};

The use of Popen is based on an answer to this question:

Read streaming input from subprocess.communicate()

The -u option is passed to python to disable output buffering.

To clarify, the issue is that the command runs a script that has many print statements and runs for several minutes so I need to send the stdout as it occurs over the socket back to the client, not after it's finished running — siddhant1999, Aug 18 '19 at 04:13
That is the kind of detail that needs to be added to the question. None of that is obvious from the current version of the question. — , Aug 18 '19 at 04:14
No need to apologize. Sometimes folks don't realize that a clarifying comment needs to be added to the question so that everyone gets to see it. — , Aug 18 '19 at 04:32
With the updated answer the ` for line in process.stdout:` loop doesn't execute until the long running python script is completed executing. Is there a way to get live logs? — siddhant1999, Aug 18 '19 at 23:49
Here's the script I'm executing to test: from time import sleep for i in range(40): print (i) sleep(0.1) — siddhant1999, Aug 19 '19 at 00:17
I modified the answer to disable `python` output buffering. See if it helps. — , Aug 19 '19 at 05:08

בנימין כהן · Answer 2 · 2019-08-19T12:37:37.763

this class will serve as a wrapper for the server

import sys


class ServerWrapper:
    def __init__(self, ws):
        self.__ws = ws
        sys.stdout = self

    def write(self, data):
        self.__ws.send(data)

    def close(self):
        sys.stdout = sys.__stdout__

you need to initialize it with your websocket.

the write function is being called every every time print is called (because we changed sys.stdout to our custom output.

then, after you finished executing the script, you can restore the standard output with close


import asyncio
import websockets
import subprocess
import sys


class ServerWrapper:
    def __init__(self, ws):
        self.__ws = ws
        sys.stdout = self

    def write(self, data):
        self.__ws.send(data)

    def close(self):
        sys.stdout = sys.__stdout__


async def time(websocket, path):
    wrapper = ServerWrapper(websocket)
    # get commands and execute them as you would normally do
    # you don't need to worry about reading output and sending it


start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Could you please show me how that would fit into my existing code — siddhant1999, Aug 18 '19 at 05:14
as I said.. I never worked with this library. then how do I get the websocket object? — בנימין כהן, Aug 19 '19 at 06:07

How to send the output of a long running Python script over a WebSocket?

2 Answers2

Linked