7

From the celery help function:

> celery worker -h

...

Embedded Beat Options:
  -B, --beat            Also run the celery beat periodic task scheduler. Please note that there must only be
                        one instance of this service. .. note:: -B is meant to be used for development
                        purposes. For production environment, you need to start celery beat separately.

This also appears in the docs.

You can also embed beat inside the worker by enabling the workers -B option, this is convenient if you’ll never run more than one worker node, but it’s not commonly used and for that reason isn’t recommended for production use:

celery -A proj worker -B

But it's not actually explained why it's "bad" to use this in production. Would love some insight.

zerohedge
  • 3,185
  • 4
  • 28
  • 63

1 Answers1

10

The --beat option will start a beat scheduler along with the worker.

But you only need one beat scheduler。

In the production environment, you usually have more than one worker running. Using --beat option will be a disaster.

For example: you have a event scheduled at 12:am each day.

If you started two beat process, the event will run twice at 12:am each day.

If you’ll never run more than one worker node, --beat option if just fine.

gushitong
  • 1,898
  • 16
  • 24
  • @zerohedge worker node = worker process, two worker node(process) usually run on different machine in the production. – gushitong Jul 18 '18 at 04:24
  • so if I just have this `celery -A myappname worker` currently, changing that to `celery -B myappname worker` shouldn't have adverse effects in production? (running my app on Heroku if that's of any relevance) – zerohedge Jul 18 '18 at 04:25
  • If you have only one worker node, `-B` will just fine. Otherwise, you need to run a individual beat process. – gushitong Jul 18 '18 at 04:30
  • that's what I'm not sure about, is that considered a single node? – zerohedge Jul 18 '18 at 04:34
  • from this [answer](https://stackoverflow.com/questions/31898311/celery-difference-between-concurrency-workers-and-autoscaling) on SO: "Let's distinguish between workers and worker processes. You spawn a celery worker, this then spawns a number of processes (depending on things like --concurrency and --autoscale, the default is to spawn as many processes as cores on the machine)" – zerohedge Jul 18 '18 at 04:38
  • If you run `celery -A app worker -c 4 --beat `, celery will start spawn one beat process to schedule event, four child processes to process the queue. This is called a worker node. (Maybe I'm wrong about `worker node = worker process`) – gushitong Jul 18 '18 at 04:53
  • sorry, I still don't understand if that's considered OK or not. is `celery -A app worker -c 4 --beat` valid on production, or not? **what is** called a worker node, the four concurrent child processes, or...? – zerohedge Jul 18 '18 at 04:55
  • 1
    When you run a celery worker, it creates one parent process to manage the running tasks. This process handles the book keeping features like sending/receiving queue messages, registering tasks, killing hung tasks, tracking status, etc. That process then spawns N number of child worker processes that actually execute the individual tasks. The number is determined by the -c argument when starting the worker. The parent process(with its all child process) is called a worker node. – gushitong Jul 18 '18 at 05:03
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/176208/discussion-between-gushitong-and-zerohedge). – gushitong Jul 18 '18 at 05:06
  • It would be great to see the end of this conversation. I think the confusion is around the celery worker spawning multiple child worker processes (based on number of CPU cores). Will these child workers all be managed by the embedded beat (if you were using embedded beat) or would you run this risk of creating duplicate tasks? Can you maybe elaborate on whether the schedule of an embedded beat will result in duplicate tasks it the are multiple child workers, but a single celery worker node? – Ryan-Neal Mes Jan 10 '19 at 17:51
  • @gushitong your chat link is not available, hence cannot get any further updates. Any updates on this? – M.A.K. Simanto Feb 12 '19 at 14:15