0

I have a fairly simple question - I am under the impression that Celery (with Redis) pushes each task to the Redis List called 'celery' by default.

I wanted to be able to see this Redis List in order to use it in other applications (to see the task lengths), but when I view my Redis Queue to see how long the 'celery' Redis List is (given that my tasks are launched in that Queue), it's not there.

Note: using Docker with Redis Image 5.0 on Port 3780 (to isolate it from the rest of my Django Application)

According to this: https://stackoverflow.com/a/43420719/10112899 I should be able to just query the length of the celery queue I specified (default = celery), but it's not there!

Here's what the Redis CLI gives me:

enter image description here

Question: Does Celery For Django and Python Not Use Redis Lists at all?

Omar Sumadi
  • 143
  • 2
  • 7

1 Answers1

1

Celery does use Redis LIST/ZSET but there're certain things to remember. If any collection is empty then Redis deletes the key so if you push a couple of entries then you should be able to identify queues.

sonus21
  • 5,178
  • 2
  • 23
  • 48
  • I actually did some more digging and found that the Redis Queue is only populating under the 'celery' list what my worker can't immediately "take in and process". I set my concurrency to 1 from 16 and launched 60 tasks (20s sleep) and I started to see the Redis Queue populate under the 'celery' list finally. On the flip side, When I set my concurrency to 16 and ran fasts (20s sleep) (60 tasks), my workers would somehow immediately ingest them, causing them to never show under the 'celery' list on the Redis Queue - even though I didn't have enough workers (60 tasks vs 16 workers) – Omar Sumadi Mar 03 '21 at 16:01
  • Also check this out: https://groups.google.com/g/celery-users/c/lLL7lyTm_UA/m/1xU8pa0qAwAJ – Omar Sumadi Mar 03 '21 at 16:08