If multiple clients make a request to one server(pytorch model), how does the problem arise in the multithread of the flask or elsewhere?
As far as I know, if running the pytorch model with multithreading using flask, the data will get tangled. If I’m wrong, please correct it.
Use a multithreaded approach for Flask as well, the request will get throttled if you just initiate a single threaded usual Flask server.
Use Gunicorn: https://gunicorn.org/ to start your Flask server, with a good number of threads, passed as a parameter value to Gunicorn.
