4

i have a simple structure:

  • proj

    • celery.py
    • tasks.py
  • run_tasks.py

celery.py:

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('proj',
         broker='amqp://',
         backend='amqp://',
         include=['proj.tasks'])

app.conf.update(
  result_expires=3600,
  task_annotations={
    'proj.tasks.add': {'rate_limit': '2/m'}
 }
)

tasks.py:

from __future__ import absolute_import, unicode_literals
from .celery import app


@app.task
def add(x, y):
   return x + y

run_tasks.py:

 from proj.tasks import *

 res = add.delay(4,4)
 a = res.get()
 print(a)

According to the parameter {'rate_limit': '2/m'} I can run the add task only 2 times a minute. But I can run it as many times as I want. What's wrong?

gonzo
  • 549
  • 1
  • 8
  • 14

1 Answers1

3

From Celery Docs:

Note that this is a per worker instance rate limit, and not a global rate limit. To enforce a global rate limit (e.g., for an API with a maximum number of requests per second), you must restrict to a given queue.

mastisa
  • 1,875
  • 3
  • 21
  • 39
  • if you do restrict it to a given queue, can you have multiple workers on that queue? – PirateApp Mar 02 '19 at 14:06
  • 1
    If I understood your meaning, restriction is on queue so (I'm not sure because I didn't test it) It's not a problem to have multi worker on that queue – mastisa Mar 03 '19 at 05:27
  • I am having a problem understanding this and unfortunately that little paragraph is the only thing that remotely tries (and fails spectacularly) to explain it in the official docs. The rate limit in ops example is set on the task, not the queue. So how do you set it on a queue? – Vigrond Jun 14 '19 at 22:42
  • 1
    As I said, I didn't test it my self, so I can't answer your question but maybe this answer can help you https://stackoverflow.com/a/28380129/8258902 – mastisa Jun 15 '19 at 04:40