FastAPI `run_in_threadpool` getting stuck

Question

I have implemented all my routes using async. And have followed all guidelines from the FastAPI documentation.

Every route has multiple DB calls, which does not have async support, so they are normal function like this

def db_fetch(query):
    # I take a few seconds to respond
    return

To avoid blocking my event loop I use fastapi.concurrancy.run_in_threadpool

Now the issue is, when a large number of requests come, my new requests are getting blocked. Even if I close the browser tab (cancel request), the entire app gets stuck till the older requests get processed.

What am I doing wrong here?

I use uvicorn as my ASGI server. I run in a kubernetes cluster with 2 replica.

Few suspects: Am I spawning too many threads? Is it some bug within uvicron? Not really sure!

It could really be the number of threads. You could try to create a dedicated ThreadPool with a defined number of threads, and after that use asyncio run_in_executor using the predefined exexutor. — Simon Hawe, Jan 31 '22 at 15:02
Makes sense. I was going for that initially, but then I was suggested to use fastapi's `run_in_threadpool` as it uses AnyIO to manage threads, and takes care of housekeeping. — Irfanuddin, Jan 31 '22 at 15:14
That is what fastapi or starlett use under the hood https://anyio.readthedocs.io/en/stable/threads.html and they explicitly point that too many threads might be an issue — Simon Hawe, Jan 31 '22 at 15:15
If I create a dedicated threadpool, won't it cause the same issue? — Irfanuddin, Jan 31 '22 at 15:18
I think you can limit the number of worker threads https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor So if you define that pool once as a global var and use it, I think you can achieve this limit. — Simon Hawe, Jan 31 '22 at 15:24
Does this answer your question? [FastAPI runs api-calls in serial instead of parallel fashion](https://stackoverflow.com/questions/71516140/fastapi-runs-api-calls-in-serial-instead-of-parallel-fashion) — Chris, Jan 22 '23 at 08:10

score 7 · Answer 1 · edited Dec 23 '22 at 06:24

It is as you've said an issue with too many threads. Under the hood, fastapi uses starlette which in turn uses anyio's to_thread.run_sync. As described here, too many threads could lead to an issue and you could shield them using a semaphore to set an upper bound on the maximum threads created. In code, that would read roughly like

# Core Library
from typing import TypeVar, Callable
from typing_extensions import ParamSpec
# Third party
from anyio import Semaphore
from starlette.concurrency import run_in_threadpool

# To not have too many threads running (which could happen on too many concurrent
# requests, we limit it with a semaphore.
MAX_CONCURRENT_THREADS = 10
MAX_THREADS_GUARD = Semaphore(MAX_CONCURRENT_THREADS)
T = TypeVar("T")
P = ParamSpec("P")


async def run_async(func: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T:
    async with MAX_THREADS_GUARD:
        return await run_in_threadpool(func, *args, **kwargs)

After adding this one, what would happen to my other requests? Now that threads are restricted, they'd all be in waiting state right? Or does starlette spawn a new threadpool? — Irfanuddin, Jan 31 '22 at 17:57

FastAPI `run_in_threadpool` getting stuck

1 Answers1

Linked