Stack Overflow
Stack Overflow

Questions tagged [celeryd]

celeryd is the common name of a Celery worker node, running as a daemon.

Celery is a widely used asynchronous job/task queue processing engine, written in Python and available under the New BSD License. It can use a large variety of message brokers like RabbitMQ, Redis, MongoDB, CouchDB, Amazon SQS and others.

More info here: http://celery.github.com/celery/getting-started/introduction.html#overview

148 questions
99
votes
5 answers

Understanding celery task prefetching

I just found out about the configuration option CELERYD_PREFETCH_MULTIPLIER (docs). The default is 4, but (I believe) I want the prefetching off or as low as possible. I set it to 1 now, which is close enough to what I'm looking for, but there's…
Henrik Heimbuerger
  • 9,924
  • 6
  • 56
  • 69
65
votes
2 answers

Setting Time Limit on specific task with celery

I have a task in Celery that could potentially run for 10,000 seconds while operating normally. However all the rest of my tasks should be done in less than one second. How can I set a time limit for the intentionally long running task without…
jimstandard
  • 1,067
  • 2
  • 10
  • 17
53
votes
3 answers

How can I recover unacknowledged AMQP messages from other channels than my connection's own?

It seems the longer I keep my rabbitmq server running, the more trouble I have with unacknowledged messages. I would love to requeue them. In fact there seems to be an amqp command to do this, but it only applies to the channel that your connection…
Will Olbrys
  • 957
  • 2
  • 9
  • 13
51
votes
9 answers

How do I restart celery workers gracefully?

While issuing a new build to update code in workers how do I restart celery workers gracefully? Edit: What I intend to do is to something like this. Worker is running, probably uploading a 100 MB file to S3 A new build comes Worker code has…
Quintin Par
  • 15,862
  • 27
  • 93
  • 146
36
votes
3 answers

How does a Celery worker consuming from multiple queues decide which to consume from first?

I am using Celery to perform asynchronous background tasks, with Redis as the backend. I'm interested in the behaviour of a Celery worker in the following situation: I am running a worker as a daemon using celeryd. This worker has been assigned two…
Jonathan Evans
  • 1,173
  • 2
  • 10
  • 22
31
votes
6 answers

How to start a Celery worker from a script/module __main__?

I've define a Celery app in a module, and now I want to start the worker from the same module in its __main__, i.e. by running the module with python -m instead of celery from the command line. I tried this: app = Celery('project',…
Fred Foo
  • 355,277
  • 75
  • 744
  • 836
25
votes
7 answers

How to restart Celery gracefully without delaying tasks

We use Celery with our Django webapp to manage offline tasks; some of these tasks can run up to 120 seconds. Whenever we make any code modifications, we need to restart Celery to have it reload the new Python code. Our current solution is to send a…
nitwit
  • 1,745
  • 2
  • 17
  • 20
21
votes
2 answers

Celery node fail, on pidbox already using on restart

I have Celery running with RabbitMQ broker. Today, I have a failure of a Celery node, it doesn't execute tasks and doesn't respond on service celeryd stop command. After few repeats, the node stopped, but on start I get this…
Jamal
  • 365
  • 2
  • 3
  • 10
20
votes
2 answers

Celery: number of workers vs concurrency

What is the difference between having: one worker with concurrency 4 or two workers with concurrency 2 each for the same queue. Thanks
Olivetree
  • 415
  • 1
  • 4
  • 8
20
votes
2 answers

Using celeryd as a daemon with multiple django apps?

I'm just starting using django-celery and I'd like to set celeryd running as a daemon. The instructions, however, appear to suggest that it can be configured for only one site/project at a time. Can the celeryd handle more than one project, or can…
Jordan Reiter
  • 20,467
  • 11
  • 95
  • 161
18
votes
3 answers

Increase celery retry time each retry cycle

I do retries with celery like in the Docs-Example: @task() def add(x, y): try: ... except Exception, exc: add.retry(exc=exc, countdown=60) # override the default and # retry in 1…
Gregor
  • 4,306
  • 1
  • 22
  • 37
17
votes
4 answers

Daemonizing celery

Following instructions found here, I copied the script from github into /etc/init.d/celeryd, then made it executable; $ ll /etc/init.d/celeryd -rwxr-xr-x 1 root root 9481 Feb 19 11:27 /etc/init.d/celeryd* I created config file /etc/default/celeryd…
Jinxcat
  • 584
  • 1
  • 3
  • 17
17
votes
1 answer

In celery, what would be the purpose of having multiple workers process the same queue?

In the documentation for celeryd-multi, we find this example: # Advanced example starting 10 workers in the background: # * Three of the workers processes the images and video queue # * Two of the workers processes the data queue with loglevel…
chaimp
  • 16,897
  • 16
  • 53
  • 86
16
votes
5 answers

how to remove task from celery with redis broker?

I Have add some wrong task to a celery with redis broker but now I want to remove the incorrect task and I can't find any way to do this Is there some commands or some api to do this ?
timger
  • 944
  • 2
  • 13
  • 31
13
votes
2 answers

How do I troubleshoot an exit timeout of celeryd when running on Heroku (error R12)?

I'm running celeryd on a Heroku dyno. When I shut it down and it has previously processed (even completed) at least one task, it doesn't shut down properly and I'm getting an error R12 (exit timeout) from Heroku. Here's how I'm running celeryd from…
Henrik Heimbuerger
  • 9,924
  • 6
  • 56
  • 69
1
2 3
…
9 10