I have a flask app with SocketIO. Due to this, I choose to run the application using the eventlet
library. Under the hood, eventlet uses green threads to achieve concurrency if I'm not mistaken.
In my app, I want to spawn a process and stream the output over web sockets. Below is a toy example
# eventlet==0.25.1
# flask==1.1.1
# flask-socketio==4.2.1
import eventlet
import subprocess
from flask import Flask, jsonify
from flask_socketio import SocketIO
app = Flask(__name__)
socketio = SocketIO(app)
socketio.init_app(app, cors_allowed_origins='*')
@app.route('/ping')
def start_ping():
eventlet.spawn_n(ping)
return jsonify(success=True)
@app.route('/hello')
def hello():
return jsonify(data='Hello World')
def ping():
proc = subprocess.Popen(
('ping', '-t', 'google.com'),
stdout=subprocess.PIPE,
)
for line in proc.stdout:
eventlet.sleep(0.5)
socketio.emit('log_receive', str(line))
if __name__ == '__main__':
socketio.run(app)
- User hits the
/ping
endpoint. - The
ping()
function is executed in a green thread which runs the ping command in a child process - lines are read from the subprocess's stdout and
emited
via web sockets eventlet.sleep(0.5)
is used to give other parts of the application a chance to run.
Question:
for line in proc.stdout:
is blocking. Until something comes through stdout, eventlet.sleep(0.5)
will not execute and therefore the rest of the application isn't given a chance to run. Thus rendering the app, unresponsive.
I came across this question on how to do non-blocking reads from subprocess.PIPE
and the suggestion is to essentially use a separate thread to do the reading.
Unfortunately, I can not use a separate thread because of the concurrent/coroutine programming model I'm restricted to because of eventlet/greenlet.
I could use fcntl() to do non-blocking reads but I'm on windows and so it isn't an option
What's an alternative to avoid having the application be at mercy of the subprocess's stdout