How to know the reason why a docker container exits?

Question

I have a Docker container running in a host of 1G RAM (there are also other containers running in the same host). The application in this Docker container will decode some images, which may consume memory a lot.

From time to time, this container will exit. I doubt it is due to out of memory but not very sure. I need a method to find the root cause. So is there any way to know what happened for this container's death?

You can check the logs for that container via `docker logs `. — techtabu, Jun 30 '16 at 02:30
but the container has exited, i guess I can not logs it any more ? — Li Bin, Jun 30 '16 at 02:32
Just tried on my machine. You can still access the logs even when the container has exited. — Samuel Toh, Jun 30 '16 at 02:36
Typically container exits when the running application dies. If the log you are seeing isn't telling you much about why it is dying then it means your application isn't logging enough. You might want to ask yourself whether Is it always crashing on the same image file? If it is, then what's wrong with it? Shouldn't be too far away from solving since you should be able to replicate it easily. — Samuel Toh, Jun 30 '16 at 02:43
@SamuelToh is there anyway to keep the container running even the app inside it dies? — Li Bin, Jun 30 '16 at 02:49
I agree with @SamuelToh too. In addition, since you have mentioned the application consumes lot of memory, you could have ran out of memory. Can you try in another machine that has more memory? Also, if you can increase memory for the particular application (like Xmx in Java), try that too. Docker daemon log might give more information, check this [link](http://stackoverflow.com/questions/30969435/where-is-the-docker-daemon-log) to see your logs. — techtabu, Jun 30 '16 at 02:49
Answer to your question is no. If container dies, app also dies. — techtabu, Jun 30 '16 at 02:50
Note that if you want to go back in and inspect a dead container, you can commit it to an image with `docker commit my_temp_image` and then run `docker run -it --entrypoint /bin/bash my_temp_image` to get a shell inside it. If there are any logs, misc files, or anything else left behind you want to inspect it'll all be there. That's assuming you have bash in your image, of course. — Giftiger Wunsch, Jan 30 '18 at 09:13
Some of you might need to note that if you `docker run --rm ...` - i.e. with "remove" option, then the container is being removed after exiting, and that's the reason you cannot get the logs. You can still access logs from a "dead" container, unless you remove (or auto-remove) it. Just get rid of the `--rm` option. — Mikaelblomkvistsson, Nov 29 '22 at 12:53

BMitch · Accepted Answer · 2022-03-05T14:48:57.477

187

Others have mentioned docker logs $container_id to view the output of the application. This would always be my first thing to check.

Next, you can run a docker inspect $container_id to view details on the state, e.g.:

    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 2,
        "Error": "",
        "StartedAt": "2016-06-28T21:26:53.477229071Z",
        "FinishedAt": "2016-06-28T21:26:53.478066987Z"
    },

The important line there is "OOMKilled" which will be true if you exceed the container memory limits and Docker kills your app. You may also want to lookup the exit code to see if it identifies a cause for the exit by your app.

Note, this only indicates if docker itself kills your process, and requires that you have set a memory limit on your container. Outside of docker, the Linux kernel can kill your process if the host itself runs out of memory. Linux often writes to a log in /var/log when this happens. With Docker Desktop on Windows and Mac, you can adjust the memory allocated to the embedded Linux VM in the docker settings.

edited Mar 05 '22 at 14:48

answered Jun 30 '16 at 02:49

BMitch

231,797
42
475
450

16

I don't understand here is since my container is gone, how the "inspect" will work ? From discussion above, once the app dies , the container will dies too. You mean restart the same image then inspect ? – Li Bin Jun 30 '16 at 02:59
14

@LiBin a container don't get wipe away when it dies, it simply gets to a halt state like status= stopped or exited. 'docker ps -a' and see for yourself – Samuel Toh Jun 30 '16 at 03:04
I was getting exit 0 every time running a memory intensive operation and OOMKilled was false. Increasing memory made it working again. – Andrei Apr 19 '18 at 13:47
1

This can happen if the Linux kernel, rather than the docker engine, kills processes in the container. You will often see that in the OS logs under /var/log on the host. – BMitch Feb 26 '19 at 16:48
2

I know it's a typo, but having the linux kernel "lol" your process seems so appropriate – Jim T Mar 05 '22 at 11:14
Man, this could've saved me 2 days of debugging! My container would just say EXITED and nothing in the actual logs. Only after looking at the inspect output did I see that it was OOMKilled. Why doesn't docker make the status OOMKilled instead of EXITED? – Ebsan Aug 11 '22 at 20:53

score 12 · Answer 2 · answered Feb 26 '19 at 14:16

You can find out whether the process inside the container was OOMkilled by reading the logs. OOMkills are initiated by the kernel so every time it happens there's a bunch of lines in /var/log/kern.log, for example:

python invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
oom_kill_process+0x22e/0x450
Memory cgroup out of memory: Kill process 31204 (python) score 1994 or sacrifice child
Killed process 31204 (python) total-vm:7350860kB, anon-rss:4182920kB, file-rss:2356kB, shmem-rss:0kB

This answer helped me find what's wrong with a container that docker would restart on exit (docker inspect does not help much here). — m90, Jul 09 '20 at 13:34

Roberto Manfreda · Answer 3 · 2020-12-03T11:24:29.837

9

While the accepted answer is the best option, sometimes it can be useful to inspect from the host the content of the journal too (on linux).

You can do that by typing:

sudo journalctl -u docker

or tailing it

sudo journalctl -u docker -f

or piping the output to less if it is too long for your terminal buffer

sudo journalctl -xn -u docker | less

edited Dec 03 '20 at 11:24

answered Mar 03 '20 at 23:22

Roberto Manfreda

2,345
3
25
39

1

That was it! Great tip. No logs anywhere from Docker, but this showed me that something killed and restarted the Daemon process. – oligofren Aug 14 '21 at 08:53

score 0 · Answer 4 · answered Aug 17 '22 at 08:08

0

docker inspect -f '{{ .State }}' $container_id and check 8th property which is ExitCode.

answered Aug 17 '22 at 08:08

Roman T

11
2

score 0 · Answer 5 · answered Nov 29 '22 at 13:00

If you cannot access the container's logs, especially when you use the docker API - e.g. because docker.errors.NotFound: 404 Client Error - the reason could be that the container has been automatically removed after exiting.

It can be caused by running the container with docker run --rm option, which is: Automatically remove the container when it exits. You will be not able to get the logs any more then.

Just get rid of --rm option and take care for removing the container manually after collecting the logs.

How to know the reason why a docker container exits?

5 Answers5

Linked