0

I run a video hosting site. Users upload videos and those videos are then processed. The video processing tasks are currently run as background tasks in celery. How can I make it so that when a user needs a video processed when my celery worker(s) are already processing the max number of tasks (also, how can I figure out how much work a worker can do? trial and error?) the user's task is put in a queue and I can display a message "You are 5th in the queue for your video to be processed" ?

Michael
  • 763
  • 1
  • 10
  • 25
  • this is already what celery does (the queue part). You just need to figure out how to determine the queue size- if you use `rabbitmq` backend look into [`pika`](https://stackoverflow.com/questions/8192584/get-queue-size-in-pika-amqp-python) – avigil Mar 20 '18 at 03:00
  • http://thepythondjango.com/scraping-10000-tweets-in-60-seconds-using-celery-rabbitmq-and-docker-clusterwith-rotating-proxy/ – Anurag Rana Mar 20 '18 at 06:32
  • **You are 5th in the queue** - You will have to ask the Rabbitmq for the length of the queue. **figure out how much work a worker can do** - the workers will accept tasks till the time they are running. You can also configure how many tasks they should do, after which they respawn. Try these: https://stackoverflow.com/questions/35231690/celery-how-to-limit-number-of-tasks-in-queue-and-stop-feeding-when-full https://stackoverflow.com/questions/5544629/retrieve-list-of-tasks-in-a-queue-in-celery – Nihal Sharma Mar 20 '18 at 06:43

0 Answers0