Having trouble killing linux processes

Question

I'm trying to restart celery after code changes by following How to restart Celery gracefully without delaying tasks. Based on this I ran:

(env1)ubuntu@ip-172-31-22-65:~/projects/tp$ ps aux|grep "celery"
ubuntu    2701  0.3  3.7 107788 37904 ?        S    12:17   0:00 /home/ubuntu/.virtualenvs/env1/bin/python3.4 /home/ubuntu/.virtualenvs/env1/bin/celery --app=tp.celery:app worker --loglevel=INFO
ubuntu    2705  0.0  3.3 107120 34132 ?        S    12:17   0:00 /home/ubuntu/.virtualenvs/env1/bin/python3.4 /home/ubuntu/.virtualenvs/env1/bin/celery --app=tp.celery:app worker --loglevel=INFO
ubuntu    2716  0.0  0.0  10460   932 pts/0    S+   12:20   0:00 grep --color=auto celery
(env1)ubuntu@ip-172-31-22-65:~/projects/tp$ sudo kill -9 2701 2705
(env1)ubuntu@ip-172-31-22-65:~/projects/tp$ ps aux|grep "celery"
ubuntu    2720 16.3  3.7 107796 37908 ?        S    12:25   0:00 /home/ubuntu/.virtualenvs/env1/bin/python3.4 /home/ubuntu/.virtualenvs/env1/bin/celery --app=tp.celery:app worker --loglevel=INFO
ubuntu    2724  0.0  3.3 107144 34084 ?        S    12:25   0:00 /home/ubuntu/.virtualenvs/env1/bin/python3.4 /home/ubuntu/.virtualenvs/env1/bin/celery --app=tp.celery:app worker --loglevel=INFO
ubuntu    2726  0.0  0.0  10460   932 pts/0    S+   12:25   0:00 grep --color=auto celery

I don't understand what is going on here. It seems like 2 new celery worker processes appear after I kill the first 2.

What am I doing wrong?

Is your celery worker supervised (for example by supervisor)? If so, the supervisor will respawn the celery worker processes after each "crash". — Balthazar Rouberol, Apr 03 '15 at 12:32
@user61629 Then why not using ``supervisorctrl update`` before a ``supervisorctrl restart celery_worker``? — Balthazar Rouberol, Apr 03 '15 at 12:41
Thank you, I used (env1)ubuntu@ip-172-31-22-65:~/projects/tp$ sudo supervisorctl restart tp-celery — user1592380, Apr 03 '15 at 12:56
The origninal process disappears , but a new process is visible. Please see above command line output. — user1592380, Apr 03 '15 at 18:10

score 1 · Accepted Answer · answered Apr 03 '15 at 14:43

I see no evidence you're doing anything wrong.

Celery is a collection of processes. The most important is a supervisor that keeps some number of workers ready to answer a request.

The worker is disposable, but the supervisor can't tell what a worker is doing. When a worker exits, the supervisor gets a signal and starts up a new one. That's it.

If you change the configuration, the supervisor can re-configure itself (IF you have it so configured!), but the workers aren't watching that configuration. They get configured only when they're born.

What you're doing with "kill" is killing those workers, and expecting the replacement workers to have the settings you want when they're born.

Asking the supervisor to stop itself and workers, and then reload, and start again, can leave several seconds where no requests get answered. That's the benefit, as far as I know, over supervisorctl.

So, there's nothing obviously wrong with anything you've told us. Of course you should expect the workers to be replaced.

Hi Chad thanks for the info.How do you get supervisor to "to stop itself and workers" ? — user1592380, Apr 03 '15 at 18:05
BTW, would you mind looking at http://stackoverflow.com/questions/29402447/how-to-set-celeryconfig-file-in-in-django . This is the problem I asked the current question about. — user1592380, Apr 03 '15 at 18:08

Having trouble killing linux processes

1 Answers1