0

I need to develop a FastAPI where the user will ask to an endpoint some info. This endpoint will ask to a webserver some info with the parameters introduced by the user, it will do some calculations (I guess everything is done in 6s) and then, it will download some big zipped files (800MB-1.5GB), unzip them, treat some info and remove them from disk.

My problem is that these files are too big to process them fast, and furthermore, I will be limited by the server to download a maximum of N files at the same time. Moreover, to download some files I'll need to wait some hours before downloading them because some server limitation. Hence, my idea is to return a message after the 6s and then process the download in a different thread.

What logic structure would you follow to deploy this FastAPI? I was considering creating two queues with celery, but I'm not sure how many workers to assign for the downloading part. Also, is it possible to call another celery task once one task is finishing? (Even if it's in a different queue?)

Learning from masters
  • 2,032
  • 3
  • 29
  • 42
  • You might find [this answer](https://stackoverflow.com/a/73240097/17865804), as well as [this answer](https://stackoverflow.com/a/73843234/17865804), helpful regarding the downloading part. – Chris Jan 06 '23 at 10:03

1 Answers1

0

Based on the official celery documentation you can follow this to specify different queues.

from kombu import Queue

app.conf.task_default_queue = 'default'
app.conf.task_queues = (
    Queue('default',    routing_key='task.#'),
    Queue('feed_tasks', routing_key='feed.#'),
)
app.conf.task_default_exchange = 'tasks'
app.conf.task_default_exchange_type = 'topic'
app.conf.task_default_routing_key = 'task.default'

You can execute the celery command by passing queue names as an argument:

celery -A app worker -l info -Q feed_tasks

Julia Meshcheryakova
  • 3,162
  • 3
  • 22
  • 42
santhosh kumar Reddy
  • 1
  • 3