Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
264
votes
2 answers

Why do we need message brokers like RabbitMQ over a database like PostgreSQL?

I am new to message brokers like RabbitMQ which we can use to create tasks / message queues for a scheduling system like Celery. Now, here is the question: I can create a table in PostgreSQL which can be appended with new tasks and consumed by the…
Yugal Jindle
  • 44,057
  • 43
  • 129
  • 197
239
votes
12 answers

Deleting all pending tasks in celery / rabbitmq

How can I delete all pending tasks without knowing the task_id for each task?
nabizan
  • 3,185
  • 5
  • 26
  • 38
212
votes
17 answers

Retrieve list of tasks in a queue in Celery

How can I retrieve a list of tasks in a queue that are yet to be processed?
bradley.ayers
  • 37,165
  • 14
  • 93
  • 99
153
votes
40 answers

Celery Received unregistered task of type (run example)

I'm trying to run example from Celery documentation. I run: celeryd --loglevel=INFO /usr/local/lib/python2.7/dist-packages/celery/loaders/default.py:64: NotConfigured: No 'celeryconfig' module found! Please make sure it exists and is available to…
Echeg
  • 2,321
  • 2
  • 21
  • 26
150
votes
11 answers

How do you unit test a Celery task?

The Celery documentation mentions testing Celery within Django but doesn't explain how to test a Celery task if you are not using Django. How do you do this?
davidmytton
  • 38,604
  • 37
  • 87
  • 93
133
votes
13 answers

How to check task status in Celery?

How does one check whether a task is running in celery (specifically, I'm using celery-django)? I've read the documentation, and I've googled, but I can't see a call like: my_example_task.state() == RUNNING My use-case is that I have an external…
Marcin
  • 48,559
  • 18
  • 128
  • 201
130
votes
8 answers

Cancel an already executing task with Celery?

I have been reading the doc and searching but cannot seem to find a straight answer: Can you cancel an already executing task? (as in the task has started, takes a while, and half way through it needs to be cancelled) I found this from the doc at…
dcoffey3296
  • 2,504
  • 3
  • 24
  • 34
126
votes
3 answers

Why use Celery instead of RabbitMQ?

From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just…
Kar
  • 6,063
  • 7
  • 53
  • 82
126
votes
2 answers

Pros and cons to use Celery vs. RQ

Currently I'm working on python project that requires implement some background jobs (mostly for email sending and heavily database updates). I use Redis for task broker. So in this point I have two candidates: Celery and RQ. I had some experience…
Max Kamenkov
  • 2,478
  • 3
  • 22
  • 19
120
votes
1 answer

Distributed task queues (Ex. Celery) vs crontab scripts

I'm having trouble understanding the purpose of 'distributed task queues'. For example, python's celery library. I know that in celery, the python framework, you can set timed windows for functions to get executed. However, that can also be easily…
Lucas Ou-Yang
  • 5,505
  • 13
  • 43
  • 62
99
votes
5 answers

Understanding celery task prefetching

I just found out about the configuration option CELERYD_PREFETCH_MULTIPLIER (docs). The default is 4, but (I believe) I want the prefetching off or as low as possible. I set it to 1 now, which is close enough to what I'm looking for, but there's…
Henrik Heimbuerger
  • 9,924
  • 6
  • 56
  • 69
94
votes
3 answers

Celery - Get task id for current task

How can I get the task_id value for a task from within the task? Here's my code: from celery.decorators import task from django.core.cache import cache @task def do_job(path): "Performs an operation on a file" # ... Code to perform the…
mattbasta
  • 13,492
  • 9
  • 47
  • 68
94
votes
3 answers

How can I run a celery periodic task from the shell manually?

I'm using celery and django-celery. I have defined a periodic task that I'd like to test. Is it possible to run the periodic task from the shell manually so that I view the console output?
Mridang Agarwalla
  • 43,201
  • 71
  • 221
  • 382
93
votes
2 answers

Celery difference between concurrency, workers and autoscaling

In my /etc/defaults/celeryd config file, I've set: CELERYD_NODES="agent1 agent2 agent3 agent4 agent5 agent6 agent7 agent8" CELERYD_OPTS="--autoscale=10,3 --concurrency=5" I understand that the daemon spawns 8 celery workers, but I'm fully not sure…
Joseph
  • 933
  • 1
  • 6
  • 4
93
votes
4 answers

Celery parallel distributed task with multiprocessing

I have a CPU intensive Celery task. I would like to use all the processing power (cores) across lots of EC2 instances to get this job done faster (a celery parallel distributed task with multiprocessing - I think). The terms, threading,…
Prometheus
  • 32,405
  • 54
  • 166
  • 302
1
2 3
…
99 100