Python multiprocessing in flask

Question

This question has probably been asked, and more than likely answered, but I don't know where to find it here.

Problem: I have a router for python's flask, that takes a while to process data for each call. I need to make each of the calls to the routes be a thread in itself so it doesn't have to wait for the requests to be loaded.

Do you want to use multi-threading for a single request? Or do you want each request to be a separate process/thread? — Brendan Abel, Feb 25 '16 at 02:03

Brendan Abel · Accepted Answer · 2018-06-21T17:03:59.067

Flask comes with a built-in development web server, but you shouldn't be using it in production.

To get cool features like separate processes for each request and static file serving, you need to run an actual web service and a WSGI service in front of your Flask application.

The Flask docs provide several examples on how to set that up. Popular Web Server/WSGI combinations are Apache/mod_wsgi and Nginx/Gunicorn, but there are many other options.

score 6 · Answer 2 · answered Jul 25 '18 at 20:24

As others have mentioned, there are specialized tools to do this (celery seems to be the best), but if someone just wants to quickly get something set up and working, here is my approach which only uses Python's multiprocessing module:

from flask import Flask
from multiprocessing import Process
import time

app = Flask(__name__)

def detachedProcessFunction(wait_time):
    i=0
    while i<wait_time:
        i = i+1
        print "loop running %d" % i
        time.sleep(1)

@app.route('/start')
def start():
    global p
    p = Process(target=detachedProcessFunction, args=(15))
    p.start()
    return render_template('layout.html')

if __name__ == '__main__':
    app.run(debug=True)

NOTE: This method won't work for running object functions (e.g. obj.objFunc()). You will get an EOFError: Ran out of input inside a class. In this case, you should create the object inside a non-object/standalone function and pass the arguments you need to create the object. For more info see here

using this , it seems the flask application still wont be available during that timeout period of 15 secs @DankMasterDan — Uddhav Mishra, Nov 20 '18 at 07:39
@Uddhav what timeout period are you referring to? I did not see any time where the flask application was unavailable.. — DankMasterDan, Nov 20 '18 at 17:14

score 5 · Answer 3 · answered Feb 25 '16 at 15:09

A really good way of setting this up would be to use "uwsgi" as your application server (and protocol) and Nginx as your front-end proxy. These are super quick, scalable, handle threading, and it is one of the Flask-recommended methods. Though the flask documentation provides the basic config, this guide is one I've used and it gives a much-more in-depth walkthrough for installation. They are using Ubuntu, but with minor changes (to the install commands) it will work on most Linux flavors.

https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-14-04

Python multiprocessing in flask

3 Answers3

Linked