37

I have a cronjob running inside a docker container that checks whether all services are running as expected. If this cronjob determines that there is a problem I would like to stop the docker container (from inside...)

Unfortunately exit just stops my cronjob script

Nils Ziehn
  • 4,118
  • 6
  • 26
  • 40
  • 1
    what image is your container based on? – Thomasleveil Jul 21 '15 at 12:24
  • 1
    it's from ubuntu:12.04 – Nils Ziehn Jul 21 '15 at 15:31
  • 1
    This seems like an [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). You might want to use a process manager like [supervisord](http://supervisord.org/) as your entrypoint to manage the multiple process and use a [supervisord-watchdog](https://github.com/Supervisor/supervisor/issues/712) event listener to terminate supervisord on a particular condition. – StockB Apr 01 '19 at 18:29
  • If you have apache running on the Foreground - just kill that process(es) (`ps -afx` to list processes, `kill PIDNUMBER` (usually `kill 1`) to kill a process), also seems to work with mariadb / `mysqld` – jave.web Oct 11 '22 at 17:06

7 Answers7

18

Basically, you need PID 1 to exit to stop the container.

Doing kill -s SIGKILL 1 won't work because PID 1 is protected.

As suggested by @Thomasleveil, you could add code such as trap "exit" SIGINT SIGTERM to the PID 1 script, which will mean the process will exit when sent a kill -s SIGINT 1. I slightly prefer this method to the one you came up with (killing the child process directly) as it gives the parent process a chance to clean up and also the parent process should be able to find the PID of the child process without awk.

The easiest way to handle this is with some sort of supervisor process, and handily Docker now provide one called "tini". Just run add the argument --init to docker run and Docker will set up tini as PID 1 in the container. A full description is here: https://docs.docker.com/engine/reference/run/#specify-an-init-process

Luke Miles
  • 941
  • 9
  • 19
Adrian Mouat
  • 44,585
  • 16
  • 110
  • 102
  • I thought of that too, but was unable to make it work. Would you have a Dockerfile for a showcase? – Thomasleveil Jul 21 '15 at 14:31
  • Err, don't you? Is there an error message? What user does the cron job run as? – Adrian Mouat Jul 21 '15 at 14:40
  • I tried with a shell script which content is `kill -s SIGKILL 1` + infinite loop/sleep and make docker run that shell script on start (as root). So I have no cron, just a simple script running ; maybe a process cannot kill itself? Also `man 2 kill` seems to indicate that the kill command refuses to send signals to the process if that process does not explicitly define a handler for that particular signal. – Thomasleveil Jul 21 '15 at 14:43
  • BTW the kill command won't know or care if the process defines a handler AFAIK. – Adrian Mouat Jul 21 '15 at 15:24
  • 2
    adding `trap "exit" SIGINT SIGTERM` to my shell script did allow me to use the `kill` command on PID `1`. I guess we cannot answser the particular case of the OP unless we know what docker image is in use – Thomasleveil Jul 21 '15 at 15:28
  • 14
    What actually worked for me is just killing every process. I have a shell script running as PID 1, which has a long running subprocess. The subprocess is not protected and just killing that subprocess allows me to exit the machine. ps x | awk {'{print $1}'} | awk 'NR > 1' | xargs kill – Nils Ziehn Jul 21 '15 at 15:30
  • @Thomasleveil Yeah, you can't do a force kill on PID 1, but it can trap and handle other signals I guess. – Adrian Mouat Jul 21 '15 at 15:33
  • 7
    @NilsZiehn you should answer your own question with details that would help others on what is the actual issue and the different ways to overcome it. Also, take a look at the `pkill` command – Thomasleveil Jul 21 '15 at 18:57
  • My entrypoint script looks like: `#!/bin/bash; trap "exit" SIGINT SIGTERM; cat` and my container is invoked using `docker run -dt --entrypoint /root/killable-entrypoint image-name:0.1` but `kill -s SIGKILL 1` still has no effect. Any advice? – StockB Apr 01 '19 at 18:55
  • @StockB As explained, PID 1 seems to be protected against `SIGKILL`. If you added the trap command, run a `SIGINT` instead. – Adrian Mouat Apr 02 '19 at 08:26
  • 1
    @AdrianMouat Neither `kill -s SIGINT 1` nor `kill -s SIGKILL 1` work. Although I've since discovered that changing to the PID to negative to kill the whole PID group seems to work e.g. `kill -s SIGINT -1` or `kill -s SIGKILL -1`. Seems contrary to your experience... I wonder if there's something different about your entrypoint script. – StockB Apr 02 '19 at 16:02
  • 2
    In case anyone ends up here, a straight forward and correct way to do this now is using tini for init -- it's built into docker. check out: https://docs.docker.com/engine/reference/run/#specify-an-init-process – Mr.Budris Mar 03 '20 at 18:39
  • Thanks Mr.Budris - I'll add it to the suggestions at the end. – Adrian Mouat Mar 04 '20 at 10:50
  • I needed to use `SIGTERM` instead of `SIGKILL` to get the `docker run --init` command to exit. I’m [told](https://stackoverflow.com/questions/16423732/linux-shell-kill-signal-sigkill-kill) that the best cross-platform and cross-library way to send this is using `kill -15 1`. The full solution for me to kill the container from inside is to specify the `--init` argument to `docker run --init` and for my process to execute the command `kill -15 1` from inside the container. – colllin Jul 12 '21 at 23:31
13

I faced the same problem and I solved it by using healthcheck feature of docker.

You can add checks to verify your services in a file like heahthcheck.sh. Then append something like following to your Dockerfile :

# Healthcheck
COPY healthcheck.sh /
RUN chmod +x /healthcheck.sh

HEALTHCHECK --interval=10s --retries=5 CMD /healthcheck.sh

This makes sure that docker checks the health of your container by running your script in specified interval and if it fails 5 times in a row, it marks the state unhealthy.

Now, in order to restart your container if the state goes unhealthy you can use autoheal like this:

docker run -d \
    --name autoheal \
    --restart=always \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -v /var/run/docker.sock:/var/run/docker.sock \
    willfarrell/autoheal

This will restart your container every time its state goes unhealthy.

Lokesh
  • 2,842
  • 7
  • 32
  • 47
5

Inspired by @JakeRobb: You can execute the docker with the command:

/bin/bash -c "while true; do sleep infinity || exit 0; done"

This will execute sleep on a PID lower than 1. Once you kill it, then the loop will exit. For that, from within the docker use:

pkill -f sleep
Yuval Atzmon
  • 5,645
  • 3
  • 41
  • 74
  • 2
    Can you clarify exactly what you mean by "execute docker with the command"? – JakeRobb Aug 21 '19 at 14:11
  • 1
    If he meant `docker exec...` it doesn't work it just freezes, maybe he meant to run this as the opening command when the container is starting, then it would work. – jave.web Oct 11 '22 at 16:40
4

I had to figure this out recently. After much frustration, what I landed on is this:

In your dockerfile, specify an ENTRYPOINT:

ENTRYPOINT ["/entrypoint.sh"]

Then, provide such a script. It doesn't need to do anything other than invoke your application. Feel free to add additional setup to the script, but be mindful that if the script does anything after it invokes your application, it could mask your application's return code. If that's relevant to you, make sure to have the script capture the return code and propagate it out as its own return code.

Note: Do not use exec to invoke your application!

The result will be that PID 1 belongs to entrypoint.sh, and your application will have some other PID. Mine tend to land on PID 9 or 10.

Then, when you need to kill your application, simply determine its PID and invoke kill -SIGKILL $PID, or pkill -SIGKILL yourapp. Your process, not being PID 1, will receive the signal and exit immediately. Your entrypoint.sh will promptly exit, because that's what you designed it to do after your process exits.

JakeRobb
  • 1,711
  • 1
  • 17
  • 32
  • This works great for killing the entrypoint, but does not allow to kill a container which might have been run without entrypoint. – StockB Apr 01 '19 at 17:11
  • Of course, but you are free to create your own Dockerfile for any container. Base it off the original Dockerfile if you have one, write and then COPY an appropriate entrypoint.sh, and then include the corresponding ENTRYPOINT. This doesn’t help you with containers already running, so `docker kill` those from outside and replace them with your updated images. – JakeRobb Apr 02 '19 at 01:06
  • @JakeRobb, thanks for the idea. I have posted an answer inspired by this answer, that allows to kill the container but without modifying the Dockerfile. – Yuval Atzmon Aug 21 '19 at 07:59
2

My example of a self-killed /usr/local/bin/docker-entrypoint.sh

#!/bin/sh
#

# Following guard prevents to stuck containers in DIND GitLab Runner
# If container still works in 300 seconds, this will kill yourself
THE_PID=$$
if [ $THE_PID -eq 1 ]; then
    # Docker does not allow kill process 1, so restart as child process
    /usr/local/bin/docker-entrypoint.sh
    exit $?
fi
(sleep 300; echo "Killing process: $THE_PID"; kill -9 $THE_PID) &


YOUR_SCRIPT_TEXT....
Maksym Anurin
  • 3,167
  • 1
  • 18
  • 13
0

I kill the "blocking process" i.e. the main process you docker is running; like so (I this case docker is running a sleep infinity command for demonstration purposes).

ps -afx | grep sleep | awk '{print $1}' | xargs kill -9

  • Thanks for sharing that idea! I modified it a little bit. I have a node.js application inside the docker container and use the Entrypoint. So, I just kill the "npm" process with that command: `ps -afx |grep npm | awk '{print $1}' | xargs kill -9` – gearcoded Jan 15 '21 at 15:26
-4

I tried to kill process 1 without success.

Give a try to the comment of @zero323 with shutdown -h now. It works fine (sorry, I can not vote for it directly since it is not in the list of answers).

peterh
  • 11,875
  • 18
  • 85
  • 108
  • 5
    This is what happens in the latest ubuntu image: > root@b968bf313300:/# shutdown -h now Failed to connect to bus: No such file or directory Failed to talk to init daemon. And it does nothing. – Kevin Buchs Aug 29 '17 at 18:44
  • 5
    Downvoted because this doesn't work. `shutdown` communicates with `initd`/`systemd`/etc, which is PID 1 in a non-container scenario. In a container, it's not present. Same goes for `reboot`. – JakeRobb Feb 20 '19 at 16:52
  • For me, using a node:10 image, i get `bash: shutdown: command not found` – andrew lorien Jul 17 '19 at 02:20