I use FastAPI for a production application that uses asyncio almost entirely except when hitting the database. The database still relies on synchronous SQLAlchemy as the async version was still in alpha (or early beta) at the time.
While our services do end up making synchronous blocking calls when it hits the database it's still wrapped in async functions. We do run multiple workers and several instances of the app to ensure we don't hit serious bottlenecks.
Concurrency with Threads
I understand that FastAPI offers concurrency using threads when using the def controller_method
approach but I can't seem to find any details around how it controls the environment. Could somebody help me understand how to control the maximum threads a process can generate. What if it hits system limits?
Database connections
When I use the async await model I create database connection objects in the middleware which is injected into the controller actions.
@app.middleware("http")
async def db_session_middleware(request: Request, call_next):
await _set_request_id()
try:
request.state.db = get_sessionmaker(scope_func=None)
response = await call_next(request)
finally:
if request.state.db.is_active:
request.state.db.close()
return response
When it's done via threads is the controller already getting called in a separate thread, ensuring a separate connection for each request?
Now if I can't limit the number of threads that are being spawned by the main process, if my application gets a sudden surge of requests won't it overshoot the database connection pool limit and eventually blocking my application?
Is there a central threadpool used by FastAPI that I can configure or is this controlled by Uvicorn?
Uvicorn
I see that Uvicorn has a configuration that let's it limit the concurrency using the --limit-concurrency 60
flag. Is this governing the number of concurrent threads created in the threaded mode?
If so, should this always be a lower than my connection pool ( connection pool + max_overflow=40)
So in the scenario, where I'm allowing a uvicorn concurrency limit of 60 my db connection pool configurations should be something like this?
engine = sqlalchemy.create_engine(
cfg("DB_URL"),
pool_size=40,
max_overflow=20,
echo=False,
pool_use_lifo=False,
pool_recycle=120
)
Is there a central threadpool that is being used in this case? Are there any sample projects that I can look at to see how this could be configured when deployed at scale.
I've used Netflix Dispatch as a reference but if there are other projects I'd definitely want to look at those.