Storing global data in a flask webserver with uwsgi

Question

I am creating my first flask project and am having difficulty sharing data between modules. I'm building a websocket server that will send tasks to worker processes on client machines that are connected.

Here is a basic setup:

/Common
    config.py
/Server
    app.py
    /modules
        /server
            common.py
            jobque.py
            scheduler.py
            server.py

app.py

import modules.server.scheduler as scheduler #scheduler must be started before the server
import modules.server.server as server
import Common.config as config

app = server.app
socketio = server.socketio

if __name__ == "__main__":
  socketio.run(app,port=config.SERVER_PORT, debug=True)

server.py

from flask import Flask, Response, request, json
from flask_socketio import SocketIO, join_room, leave_room, emit

import modules.server.common as Common

app = Flask(__name__)
socketio = SocketIO(app)

...

@socketio.event
def requestwork(data=None):
    #do some stuff by accessing variable Common.JOB_QUE
    job_que = Common.JOB_QUE
    ...

scheduler.py

from apscheduler.schedulers.background import BackgroundScheduler

import modules.server.jobque as jobque
import modules.server.common as Common

def watch_jobs():
    work = jobque.que_work() #calls a module that does a sql call and returns data
    
    #There is some extra logic I'm skipping that only adds rows that don't already exist in the job que, 
    #but I'll skip that for simplicity
    Common.JOB_QUE = work

scheduler = BackgroundScheduler()
scheduler.add_job(func=watch_jobs, trigger="interval", seconds=30)
scheduler.start()

So every 30 seconds, the app checks sql for work to be done and stores it in the JOB_QUE, which is a variable imported in both the server and the scheduler

When I run this as a flask development server, everything works expected, both modules can read and write to the same JOB_QUE and see the changes each module made.

But when running my app through uwsgi, this is no longer the case

uwsgi --http :5000 --gevent 1000 --http-websockets --master --wsgi-file app.py --callable app

When run this way, the scheduler will write to the JOB_QUE but when the server goes to read it to return work to the worker, the JOB_QUE is empty. I understand this probably has to do with uwsgi having multiple worker processes (although I tried forcing it to run with only one worker and had the same results) or maybe something to do with multi threading, but I'm not sure how to solve this without resorting to a database (which seems like overkill for this) or serializing the JOB_QUE to a file.

I read something about making sure the global variable is only imported once in the top level of the app. I tired importing it in app.py and having the other modules import that from app.py but I was not able to get the import to work.

What is an acceptable way to make this work?

Storing global data in a flask webserver with uwsgi

0 Answers0