python flask run a slow function off a webhook that requires a response in 5 seconds

Question

Running on Windows.

This is similiar to the question python flask - run script after processing the request however the answers to that one were insufficient.

Given the code:

app = Flask(__name__)

@app.route('/',methods=['POST'])
def processWebhook():
    time.sleep(30) # slow_function()
    return "received"

if __name__ == '__main__':
   app.run()

I need the slow_function to run for each POST, however it can't stop the response being returned within 5 seconds (say the function is time.sleep(30)).

Also, each slow_function cannot start until the one before has finished.

Is there a way I can add "jobs" to a second thread that execute linearly rather than just calling slow_function?

Answer 1

You simply cannot run long running processes in a web server, they are build to do the opposite and trying to force them to do it leads to ruin in the end.

Your best bet is to offload the long running function to a separate python process. Have a look at celery: http://www.celeryproject.org/ You do something homegrown using Redis / RabbitMq and consuming program.

Your requirement that only one function runs at a time should be achievable with all three.

Note that the number of moving parts increases if you use this solution, instead of one process, you have at least three (webserver, redis /rabbitmq and your consumer). Since orchestrating that is annoying, you might want to use docker (and docker-compose) to simplify the orchestration, e.g. restarting a component or monitoring it.