10

I'm implementing a cache server that uses a celery task to update the cache in background. There is only one task that I call it with different arguments (cache keys).

Since after connecting this server to my main production server it will receive tens of requests per second for the same cache key I want to make sure there are never more than one of the update tasks with the same cache key inside celery queue (working as a queue and a set at the same time).

I thought of using a redis set to make sure of that before running the task but I'm looking for a better way.

Sina
  • 183
  • 1
  • 10
  • 1
    Is your requirement really "only one task with the same cache key inside celery QUEUE" or is it "only one task with the same cache key running (updating cache) concurrently"? – Muhammad Tahir May 04 '16 at 11:24
  • @MuhammadTahir only one task with the same cache key inside celery QUEUE – Sina May 04 '16 at 12:01
  • 1
    See if this works for you http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html#ensuring-a-task-is-only-executed-one-at-a-time. This will work as it is for the second case that I mentioned but it can be modified for your version (the first I mentioned) too. – Muhammad Tahir May 04 '16 at 12:12
  • Possible duplicate of [Running "unique" tasks with celery](https://stackoverflow.com/questions/4095940/running-unique-tasks-with-celery) – funky-future Oct 05 '17 at 00:14
  • @funky-future Possibly yes. I will try to find the code I was writing at the time to check if there is any difference in requirements for both questions. Though I remember I solved my problem using this https://github.com/cameronmaske/celery-once . – Sina Oct 14 '17 at 08:08

1 Answers1

3

There is only one way, implement your own lock mechanism. The official doc has a nice example page.. The only limit is your imagination.

Hope this helps.

Mauro Rocco
  • 4,980
  • 1
  • 26
  • 40