38

I read this in the celery documentation for Task.rate_limit:

Note that this is a per worker instance rate limit, and not a global rate limit. To enforce a global rate limit (e.g., for an API with a maximum number of requests per second), you must restrict to a given queue.

How do I put a rate limit on a celery queue?

Boris Verkhovskiy
  • 14,854
  • 11
  • 100
  • 103
Vikash Singh
  • 13,213
  • 8
  • 40
  • 70
  • Maybe this article will help you implement this yourself https://callhub.io/distributed-rate-limiting-with-redis-and-celery/ – Boris Verkhovskiy Apr 03 '20 at 15:10
  • i found the [`celery.app.task.Task` code documentation (v5.0.5)](https://github.com/celery/celery/blob/v5.0.5/celery/app/task.py#L183) for rate limiting helpful in understanding its use. **note that this is specific to a _task_** and not a queue, like the OP originally asked about. but as i ended up here after visiting celery's documentation trying to figure out how it worked, figured it may help others. – cbrendanprice Jan 22 '21 at 18:22

3 Answers3

20

Turns out it cant be done at queue level for multiple workers. IT can be done at queue level for 1 worker. Or at queue level for each worker.

So if u say 10 jobs/ minute on 5 workers. Your workers will process upto 50 jobs per minute collectively.

So to have only 10 jobs running at a time you either chose one worker. Or chose 5 workers with a limit of 2/minute.

Update: How to exactly put the limit in settings/configuration:

task_annotations = {'tasks.<task_name>': {'rate_limit': '10/m'}}

or change the same for all tasks:

task_annotations = {'*': {'rate_limit': '10/m'}}

10/m means 10 tasks per minute, /s would mean per second. More details here: Task annotations setting

Vikash Singh
  • 13,213
  • 8
  • 40
  • 70
  • 7
    How about some specifics - e.g. how exactly do you tell a worker to do only "2/minute"? – spookylukey May 19 '15 at 15:14
  • @spookylukey see my answer below – c-a Mar 07 '17 at 15:50
  • 2
    After searching for the answer of this question I found that rate-limiting is possible only on task level. Celery does not support queue level rate-limiting. – Seunghoon Oct 30 '17 at 05:33
  • so is there a way to get the number of workers dynamically and then a dynamic rate limit? – PirateApp Apr 03 '18 at 04:30
  • Does this still hold in 2019? – PirateApp Mar 02 '19 at 07:12
  • 7
    For 2019 I have created an example of how to integrate rate limiting into your tasks when using Celery + Django + Redis (after giving up on Celerys lack of rate limiting features or documentation, which is probably on purpose due to differing app requirements) : https://gist.github.com/Vigrond/2bbea9be6413415e5479998e79a1b11a – Vigrond Jun 16 '19 at 02:01
  • 2
    I took @Vigrond's solution and extended it over here: https://stackoverflow.com/a/66161773/64911 – mlissner Feb 11 '21 at 19:54
0

hey I am trying to find a way to do rate limit on queue, and I find out Celery can't do that, however Celery can control the rate per tasks, see this:

http://docs.celeryproject.org/en/latest/userguide/workers.html#rate-limits

so for a workaround, maybe you can set up one tasks per queue(which makes sense in a lot of situations), and put the limit on task.

Marshall X
  • 784
  • 1
  • 5
  • 17
  • 1
    This `rate_limit` is also per worker not per task. If you have two workers consuming that task the rate limit is effectively doubled and so on. – Boris Verkhovskiy Apr 03 '20 at 15:09
-2

You can set this limit in the flower > worker pane. there is a specified blank space for entering your limit there. The format that is suggested to be used is also like the below:

The rate limits can be specified in seconds, minutes or hours by appending “/s”, >“/m” or “/h” to the value. Tasks will be evenly distributed over the specified >time frame.

Example: “100/m” (hundred tasks a minute). This will enforce a minimum delay of >600ms between starting two tasks on the same worker instance.

Community
  • 1
  • 1
Shayan
  • 125
  • 2
  • 13
  • 2
    You can set rate-limit on only task level on Flower. Celery does not support queue level rate-limiting. – Seunghoon Oct 30 '17 at 05:37