Flask socketio multithread dump dictionary to queue and send to clients

Question

Goal is simply to be able to make a thread queue of dictionaries and report them to client.

EDIT This is different of Flask throwing 'working outside of request context' when starting sub thread because:

It is not done in a route function, it is done in socketio.start_background_task

The only socketio code takes place in context, with the socketio.emit we are sending a dictionary.

Strategy: There are 2 different taks to perform in server side, for each build a thread, then in another socketio thread collect the results which are in a thread safe queue FIFO of dictionaries.

Then send these dictionaries to client and wait for each acknowledge.

So now the issue is reduced to solve: RuntimeError: Working outside of request context.

from flask import Flask, flash, request, redirect, render_template, Response, escape, jsonify, url_for, session, copy_current_request_context
#socketio
from flask_socketio import SocketIO, send, emit, join_room, leave_room, close_room, rooms, disconnect
import threading
from threading import Thread, Event, Lock
import queue
import random

def ack(value):
    if value != 'pong':
        logger.info('unexpected return value')

def fn_i():
    global q
    while True:
        time.sleep(1)
        q.put({'key_i':random.random()})
        return q

def fn_ii():
    global q
    while True:
        time.sleep(10)
        q.put({'key_ii':random.random()})
        return q

app = Flask(__name__)
socketio = SocketIO(app, async_mode=async_mode)
thread1=None
thread2=None
collector_thread=None
q = queue.Queue()
thread_lock = Lock()

def background_thread_collector():
    global thread1
    global thread2
    global q

    thread1 = threading.Thread(target=fn_i)
    thread1.start() 

    thread2 = threading.Thread(target=fn_ii)
    thread2.start() 

    """Example of how to send server generated events to clients."""
    while True:
        time.sleep(0.2)
        while not q.empty():
            socketio.emit('my_response',
                          q.get(), #{'data': 'Server generated event', 'count': count},
                          namespace='/test',
                          broadcast=True,
                          callback=ack
                         )

@app.route('/')
def index():
    return render_template('index.html', async_mode=socketio.async_mode)

@socketio.on('connect', namespace='/test')
def test_connect():
    global collector_thread
    logger.info(' Client connected ' + request.sid)
    with thread_lock:
        if collector_thread is None:
            collector_thread = socketio.start_background_task(background_thread_collector)            
    emit('my_response', {'data': 'Connected', 'count': 0})
if __name__ == '__main__':
    socketio.run(app, 
                host='localhost',
                 port=10000, 
                 debug=False) #True sends some exceptions and stops)

Cheers

Possible duplicate of [Flask throwing 'working outside of request context' when starting sub thread](https://stackoverflow.com/questions/9931738/flask-throwing-working-outside-of-request-context-when-starting-sub-thread) — Nir Alfasi, Oct 07 '19 at 22:51
Have just tried their solution and it is different, at least by adding ```with app.test_request_context():``` you get ```AttributeError: 'Request' object has no attribute 'sid'``` If you believe you are right it would be welcome if you answer with code. In this example you are not using an event, you are sending to server on an bgthread basis. — Francisco Costa, Oct 08 '19 at 00:00
Also ```@copy_current_request_context``` not working well in this case. Since i am passing a global variable queue should'nt it be available also in context, if it is global accordingly. If someone answers with that code working it would be appreciated of course. Cheers. — Francisco Costa, Oct 08 '19 at 00:09

score 0 · Answer 1 · answered Oct 09 '19 at 21:35

This should be handled better by Flask-SocketIO, but the problem is that you are trying to use a callback on an emit that is set to broadcast to all clients:

            socketio.emit('my_response',
                          q.get(), #{'data': 'Server generated event', 'count': count},
                          namespace='/test',
                          broadcast=True,
                          callback=ack
                         )

Remove the callback and the emit should work just fine.

Francisco Costa · Accepted Answer · 2019-10-11T07:42:10.833

Using threading queues with Flasksocketio is not straightforward because requires to handle the apps context among other things, for purpose of refreshing a server log file in client side found it was easier to simply use javascript in this case and the file can be updated accordingly. Even the former code with suggested alterations was not working because of apps context and nevertheless there are multiple blogs or stackoverflow which approach subject in none have found a complete working solution except implementing like explain because any other answer requires complete code which is working, and hence since able to implement here considering this is the accepted answer, Cheers.

Flask socketio multithread dump dictionary to queue and send to clients

2 Answers2