1

I'm trying to build a project with a docker compose container stack and celery as a task scheduler.

Is there any possibility to gracefully shutdown the active celery tasks when the main worker process (PID: 1) received the SIGINT signal of the docker compose down command?

Maybe someone here got experience with that kind of problem.

Thanks is advance.

kamaru510
  • 25
  • 3
  • It is supported automatically.. what's the docker command you are using for running the worker? – ItayB Oct 05 '22 at 04:05
  • @ItayB No it is definitly not. I'm starting the container stack with docker compose up . When I'm executing a docker compose down while the celery_worker still got active tasks it will exit with an error code 137 after the stop_grace_period setting defined in the docker compose file (default 10 seconds). When there are no tasks active while running the compose down cmd everything shuts down correctly and exits with code 1 – kamaru510 Oct 05 '22 at 07:33
  • You didn't answer my question, are you using a script file or celery command? – ItayB Oct 05 '22 at 09:39
  • Oh sorry I did not noticed that. I'm using a celery command to start the worker – kamaru510 Oct 05 '22 at 11:01
  • 1
    https://stackoverflow.com/a/67345613/1011253 – ItayB Oct 05 '22 at 14:06
  • Did you read this? – ItayB Oct 05 '22 at 14:06
  • Yes I already read this post. It is quite helpful to understand how celery behaves on a shutdown signal. I already tried to implement a own signal handler but did not get a result that satisfies me. I'll give it another try considering this https://www.youtube.com/watch?v=mlK_DikhNCA – kamaru510 Oct 07 '22 at 09:23
  • https://docs.celeryq.dev/en/stable/userguide/workers.html#stopping-the-worker – ItayB Oct 07 '22 at 10:11
  • From the link above you can see that the worker support the SINGTERM already – ItayB Oct 07 '22 at 10:11
  • Yes thats true and the signal does work. When receiving the SIGTERM I want to abort all currently active tasks so that the shutdown process does no run into the timeout of 10 seconds (depending on what configured in the setting 'stop_grace_period' in the docker compose file) because after the timeout a SIGKILL gets send to the process. The problem is now i can not inspect the active celery tasks with app.control.inspect().active() in the worker_shutting_down signal handler. I guess that has to do with the unsubscription from all the queues when receiving a SIGTERM – kamaru510 Oct 07 '22 at 11:41
  • I think that the "gracefull" word is wrong then, or I am missing something. Gracefully means that you are waiting for the running tasks to finish and not killing them by yourself – ItayB Oct 07 '22 at 13:16
  • You're right about that, but the long running tasks get killed anyway after the timeout of the docker software or the OS (in case of a shutdown). So that is the reason why I want a task to execute a couple statements when the workers are initiating the shutdown – kamaru510 Oct 10 '22 at 10:56
  • So this is a different question. Why do you care about those? Isn't that OK that the failing tasks will run again when a new worker will start? – ItayB Oct 10 '22 at 15:54
  • 1
    I care about those because some of my celery tasks start a subprocess. The state of these subprocesses is stored a database. These processes need a gracefull shutdown to write the database again that the process is not running anymore. I have an idea to solve that, need to try it the next days – kamaru510 Oct 13 '22 at 14:55
  • 1
    Have just written it, take a look https://stackoverflow.com/a/74629396/6550449 – Taras Mykhalchuk Nov 30 '22 at 14:57

0 Answers0