Docker - Cannot remove dead container

Question

I am unable to remove the dead container, it appears again after i restart the Docker service.

docker ps -a
CONTAINER ID         STATUS          
11667ef16239         Dead

Then

docker rm -f 11667ef16239

Then, when I ran the docker ps -a, no docker containers showing.

docker ps -a
CONTAINER ID         STATUS

However, when I restart the docker service:

service docker restart

And run the docker ps -a again:

docker ps -a
CONTAINER ID         STATUS          
11667ef16239         Dead

Hi, this is still a current issue, but the currently accepted answer is no longer valid, and pretty risky. There's the umount answer, which is a good one, so it would be fair, if you accept that one if possible — Daniel F, Jan 12 '16 at 04:45
The ``umount`` answer is not viable anymore, since it does not allow unmounting if the resource is busy. And similarly, killing the responsible pid does not work either. — runr, Oct 21 '19 at 13:44

sk8terboi87 ツ · Answer 1 · 2017-01-20T09:35:14.763

64

You can also remove dead containers with this command

docker rm $(docker ps --all -q -f status=dead)

But, I'm really not sure why & how the dead containers are created. This error seems related https://github.com/typesafehub/mesos-spark-integration-tests/issues/34 whenever i get dead containers

[Update] With Docker 1.13 update, we can easily remove both unwanted containers, dangling images

$ docker system df #will show used space, similar to the unix tool df
$ docker system prune # will remove all unused data.

edited Jan 20 '17 at 09:35

answered Dec 30 '15 at 04:37

sk8terboi87 ツ

3,396
3
34
45

@Akvel It works without them. Its already dead, so no dependent processes. – sk8terboi87 ツ Jan 12 '16 at 05:29
7

`docker system prune` did more than expected but solved the problem. As stated in the OP, `docker rm --force` does not work. – rdupz Apr 02 '17 at 14:40
2

As hinted by @rdupz, `docker system prune` **is dangerous**, see [this comment](https://stackoverflow.com/questions/32723111/how-to-remove-old-and-unused-docker-images#comment77698372_32723127) about how it removes even named volumes when with `-a`. It also removes networks, which isn't usually what you want. Better to use `docker image prune` and `docker volume prune` in 1.13+. – RichVel Aug 26 '17 at 06:43
This command is giving syntax error. I think you misplaced the force `-f` param – VPaul Sep 05 '19 at 15:25

score 53 · Accepted Answer · answered Jun 12 '15 at 01:52

53

Most likely, an error occurred when the daemon attempted to cleanup the container, and he is now stuck in this "zombie" state.

I'm afraid your only option here is to manually clean it up:

$ sudo rm -rf /var/lib/docker/<storage_driver>/11667ef16239.../

Where <storage_driver> is the name of your driver (aufs, overlay, btrfs, or devicemapper).

answered Jun 12 '15 at 01:52

icecrime

74,451
13
99
111

is the functional equivalent to this action possible by issuing a docker command instead of resorting to sudo rm ... for those like me who do not have sudo powers on our prod box – Scott Stensland Oct 10 '18 at 15:12
1

You may also want to try the opposite: `zfs create` that volume so Docker can be happy it removed it. – tadman Aug 24 '21 at 03:57
May be needed to restart docker service also after removing the files. – Mauro Baraldi Oct 04 '21 at 12:02
Specifically: Creating a new ZFS dataset solved this.. there may be more than one required per container – Mitchell Currie Mar 04 '23 at 22:03

Zgr3doo · Answer 3 · 2015-08-25T14:52:44.970

45

Actually things changed slightly these days in order to get rid of those dead containers you may try to unmount those blocked filesystems to release them

So if you get message like this

Error response from daemon: Cannot destroy container elated_wozniak: Driver devicemapper failed to remove root filesystem 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3: Device is Busy

just run this

umount /var/lib/docker/devicemapper/mnt/656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3

and you can normally remove container after that

edited Aug 25 '15 at 14:52

answered Aug 25 '15 at 14:45

Zgr3doo

1,765
17
20

This one feels like the best, most secure one! – Daniel F Jan 12 '16 at 04:47
I just rebooted the system and wen I logged back in.. the containers where gone. – superhero Nov 02 '16 at 10:25
This worked for me, though I had to restart the docker service for the output of `docker ps -a` to actually be empty. – Daniel Quinn Jan 07 '17 at 13:06
1

Two years later and this post is still relevant. Just spent two hours trying to find a solution and then came across this: https://github.com/moby/moby/issues/3786#issuecomment-33471966. To unmount all devicemapper containers use `umount $(grep 'devicemapper' /proc/mounts | awk '{print$2}' | sort -r)` – AlexanderF Sep 19 '17 at 04:30

score 30 · Answer 4 · answered Sep 06 '17 at 07:03

I had the following error when removing a dead container (docker 17.06.1-ce on CentOS 7):

Error response from daemon: driver "overlay" failed to remove root filesystem for <some-id>: 
remove /var/lib/docker/overlay/<some-id>/merged: device or resource busy

Here is how I fixed it:

1. Check which other processes are also using docker resources

$ grep docker /proc/*/mountinfo

which outputs something like this, where the number after /proc/ is the pid:

/proc/10001/mountinfo:179...
/proc/10002/mountinfo:149...
/proc/12345/mountinfo:159 149 0:36 / /var/lib/docker/overlay/...

2. Check the process name of the above pid

$ ps -p 10001 -o comm=
dockerd
$ ps -p 10002 -o comm=
docker-containe
$ ps -p 12345 -o comm=
nginx   <<<-- This is suspicious!!!

So, nginx with pid 12345 seems to also be using /var/lib/docker/overlay/..., which is why we cannot remove the related container and get the device or resource busy error. (See here for a discussion on how nginx shares the same mount namespace with docker containers thus prevents its deletion.)

3. Stop nginx and then I can remove the container successfully.

$ sudo service nginx stop
$ docker rm <container-id>

thank you sir for the nginx info, I had the same problem, it pissed me of so many times, I also had the same problem with the DNS process (named), had to: systemctl restart named — user1928596, Sep 19 '17 at 05:16

Wouter · Answer 5 · 2016-10-14T08:26:50.420

I got the same issue and both answers did not help.

What helped for me is just creating the directories that are missing and them remove them:

mkdir /var/lib/docker/devicemapper/mnt/656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3
mkdir /var/lib/docker/devicemapper/mnt/656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3-init
docker rm 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3

Lukasz Dynowski · Answer 6 · 2016-12-07T12:09:21.140

Removing container by force worked for me.

docker rm -f <id_of_the_dead_container>

Notes:

Be aware that this command might throw this error Error response from daemon: Driver devicemapper failed to remove root filesystem <id_of_the_dead_container>: Device is Busy

The mount of your's dead container device mapper should be removed despite this message. That is, you will no longer access this path:

/var/lib/docker/devicemapper/mnt/<id_of_the_dead_container>

score 9 · Answer 7 · answered Aug 31 '18 at 17:00

Tried all of the above (short of reboot/ restart docker).

So here is the error om docker rm:

$ docker rm 08d51aad0e74
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 08d51aad0e74060f54bba36268386fe991eff74570e7ee29b7c4d74047d809aa: remove /var/lib/docker/devicemapper/mnt/670cdbd30a3627ae4801044d32a423284b540c5057002dd010186c69b6cc7eea: device or resource busy

Then I did a the following:

$  grep docker /proc/*/mountinfo | grep 958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac
/proc/20416/mountinfo:629 574 253:15 / /var/lib/docker/devicemapper/mnt/958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,relatime shared:288 - xfs /dev/mapper/docker-253:5-786536-958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota

This got be the PID of the offending process keeping it busy - 20416 (the item after /proc/

So I did a ps -p and to my surprise find:

[devops@dp01app5030 SeGrid]$ ps -p 20416
  PID TTY          TIME CMD
20416 ?        00:00:19 ntpd

A true WTF moment. So I pair problem solved with Google and found this: Then found this https://github.com/docker/for-linux/issues/124

Turns out I had to restart ntp daemon and that fixed the issue!!!

You saved my day. I was struggled with issue. Even after restarting the docker daemon, the problem was not solved. I believe this answer should be accepted. Thanks. — sat, Jan 04 '20 at 07:53

score 8 · Answer 8 · answered May 14 '19 at 04:50

8

For Deleting all dead container docker rm -f $(docker ps --all -q -f status=dead)
For deleting all exited container docker rm -f $(docker ps --all -q -f status=exited)

As I have -f is necessary

answered May 14 '19 at 04:50

Pranoy Sarkar

1,965
14
31

Error: No such container: 2787a08aaf3c Doesn't work, it shows: Error: No such container: 2f13ced716af Error: No such container: 48d43ea0e93f Error: No such container: ab7720dd4392 Error: No such container: c81751444fad – juliangonzalez Jun 04 '20 at 12:39
1

Something is up with my system. You saved me a second time within two days. Haha – Sebastian Feb 14 '21 at 17:29

score 6 · Answer 9 · answered Sep 06 '17 at 10:40

Try running the following commands. It always works for me.

# docker volume rm $(docker volume ls -qf dangling=true)
# docker rm $(docker ps -q -f 'status=exited')

After execution of the above commands, restart docker by,

# service docker restart

user1179442 · Answer 10 · 2018-03-09T05:55:37.870

6

grep 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3 /proc/*/mountinfo

then find the pid of 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3and and kill it

edited Mar 09 '18 at 05:55

answered Dec 25 '17 at 02:26

user1179442

473
5
16

where the number after `/proc/` is the pid – Belter Jan 12 '18 at 08:35

score 6 · Answer 11 · edited Oct 10 '18 at 14:49

6

I have tried the suggestions above but didn't work.

Then

I try : docker system prune -a, it didn't work the first time
I reboot the system
I try again docker system prune -a. This time it works. It will send a warning message and in the end ask "Are you sure you want to continue? y/n? . Ans:y . It will time a time and in the end the dead containers are gone.
Verify with docker ps -a

IMPORTANT - this is the nuclear option as it destroys all containers + images

edited Oct 10 '18 at 14:49

Scott Stensland

26,870
12
93
104

answered Jun 01 '18 at 14:49

Nella

61
1
2

This worked for me. Thank you! I also had to do a system reboot. – pandasCat Aug 10 '18 at 16:13

score 4 · Answer 12 · answered Aug 27 '19 at 05:45

4

for Windows:

del D:\ProgramData\docker\containers\{CONTAINER ID}
del D:\ProgramData\docker\windowsfilter\{CONTAINER ID}

Then restart the Docker Desktop

answered Aug 27 '19 at 05:45

John Jang

2,567
24
28

score 2 · Answer 13 · answered Jun 07 '18 at 09:20

2

Try kill it and then remove >:) i.e.
docker kill $(docker ps -q)

answered Jun 07 '18 at 09:20

pbaranski

22,778
19
100
117

score 2 · Answer 14 · edited Nov 17 '18 at 14:19

Try, It worked for me:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
4f13b53be9dd        5b0bbf1173ea        "/opt/app/netjet..."   5 months ago        Dead                                    appname_chess

$ docker rm $(docker ps --all -q -f status=dead)
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 4f13b53be9ddef3e9ba281546aef1c544805282971f324291a1dc91b50eeb440: failed to remove device 487b4b73c58d19ef79201cf6d5fcd6b7316e612e99c14505a6bf24399cad9795-init: devicemapper: Error running DeleteDevice dm_task_run failed

su
cd /var/lib/docker/containers
[root@localhost containers]#  ls -l
total 0
drwx------. 1 root root 312 Nov 17 08:58 4f13b53be9ddef3e9ba281546aef1c544805282971f324291a1dc91b50eeb440
[root@localhost containers]# rm -rf 4f13b53be9ddef3e9ba281546aef1c544805282971f324291a1dc91b50eeb440
systemctl restart docker

score 2 · Answer 15 · answered Apr 28 '19 at 16:10

2

Try this it worked for me:

docker rm -f <container_name>

eg. docker rm -f 11667ef16239

answered Apr 28 '19 at 16:10

Adel Kedjour

21
1

Raskul · Answer 16 · 2023-04-01T10:39:51.543

2

For future users I wanna share my experience: (2023)

To see all containers I ran

docker container ls -a

then to kill containers I ran:

docker rm -f YOUR_CONTAINER_ID

for example:

docker rm -f d389s8dffe

edited Apr 01 '23 at 10:39

answered Jun 27 '22 at 04:35

Raskul

1,674
10
26

score 1 · Answer 17 · answered Oct 16 '18 at 14:11

There are a lot of answers in here but none of them involved the (quick) solution that worked for me.

I'm using Docker version 1.12.3, build 6b644ec.

I simply ran docker rmi <image-name> for the image from whence the dead container came. A docker ps -a then showed the dead container missing completely.

Then, of course, I just re-pulled the image and ran the container again.

I have no idea how it found itself in this state but so it is...

score 1 · Answer 18 · edited Jun 18 '20 at 19:51

1

The best way to get rid of dead container processes is to restart your docker service. I was unable to remove a container as it was stuck in restarting status, I just restarted the docker service and it worked for me.

edited Jun 18 '20 at 19:51

Federico Pettinella

1,471
1
11
19

answered Jun 18 '20 at 12:19

user11291930

11
2

score 1 · Answer 19 · answered Sep 16 '20 at 12:06

1

In my case, I had to remove it with

rm -r /var/lib/docker/containers/<container-id>/

and it worked. Maybe that's how you solve it in docker version ~19. My docker version was 19.03.12,

answered Sep 16 '20 at 12:06

Kirtan Gajjar

68
2
11

Cognitiaclaeves · Answer 20 · 2016-03-21T23:44:12.803

Running on Centos7 & Docker 1.8.2, I was unable to use Zgr3doo's solution to umount by devicemapper ( I think the response I got was that the volume wasn't mounted/found. )

I think I also had a similar thing happen with sk8terboi87 ツ 's answer: I believe the message was that the volumes couldn't be unmounted, and it listed the specific volumes that it tried to umount in order to delete the dead containers.

What did work for me was stopping docker first, and then deleting the directories manually. I was able to determine which ones they were by the error output of previous command to delete all the dead containers.

Apologies for the vague descriptions above. I found this SO question days after I handled the dead containers. .. However, I noticed a similar pattern today:

$ sudo docker stop fervent_fermi; sudo docker rm fervent_fermi fervent_fermi
Error response from daemon: Cannot destroy container fervent_fermi: Driver devicemapper failed to remove root filesystem a11bae452da3dd776354aae311da5be5ff70ac9ebf33d33b66a24c62c3ec7f35: Device is Busy
Error: failed to remove containers: [fervent_fermi]

$ sudo systemctl docker stop
$ sudo rm -rf /var/lib/docker/devicemapper/mnt/a11bae452da3dd776354aae311da5be5ff70ac9ebf33d33b66a24c62c3ec7f35
$

I did notice, when using this approach that docker re-created the images with different names:

a11bae452da3     trend_av_docker   "bash"   2 weeks ago    Dead    compassionate_ardinghelli

This may have been due to the container being issued with restart=always, however, the container ID matches the ID of the container that previously used the volume that I force-deleted. There were no difficulties deleting this new container:

$ sudo docker rm -v compassionate_ardinghelli
compassionate_ardinghelli

Alternative 2: rework this into its own, standalone answer. It's OK to reference other answers when doing that - but be specific about which answer you're talking about, e.g. by author's name. Starting with "I can't comment yet..." is pretty much guaranteed to get your post deleted. If you have something helpful to add as an answer - just do it! (And get out there and get 9 more rep!) Good luck. — Mogsdad, Mar 21 '16 at 15:48

score 0 · Answer 21 · answered Sep 27 '18 at 09:47

Try this it worked for me on centos 1) docker container ls -a gives you a list of containers check status which you want to get rid of 2) docker container rm -f 97af2da41b2b not a big fan force flag but does the work to check it worked just fire the command again or list it. 3) continue till we clear all dead containers

score 0 · Answer 22 · answered May 13 '21 at 15:45

0

I restarted the Docker Agent on Windows bottom right corner, docker iCon, Restart Docker. Then I ran docker ps -a and saw the container still sitting there. I then ran docker rm -f container id and that worked for me.

answered May 13 '21 at 15:45

Jason

510
5
27

score 0 · Answer 23 · answered May 15 '21 at 21:19

0

If you are a Mac/Windows user and have Docker Desktop, you can just simply restart your Docker Desktop and in the UI you can simply remove the container without any complications.

answered May 15 '21 at 21:19

Michal Moravik

1,213
1
12
23

score 0 · Answer 24 · answered Nov 04 '22 at 17:38

In my case there was no actually mounted points inside docker container. Error raised after some ifup/down game inside privileged ct. The usual way of removing container didn’t work too uprising the similar error:

user@server:~$ docker rm -f tklc.dead
Error response from daemon: containers 896c259600c951323d5db4438d8faaa0de9de4b0cf0b998ab0896e82ec1b3769: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom: operation not permitted

Nor worked any attempt to unmount any links or mount points inside. Nor to find and stop any processes associated with dead container. Nor worked any attempts to remove files directly.

user@server:~$ sudo rm -rf /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/
rm: cannot remove '/var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom': Operation not permitted
rm: cannot remove '/var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/random': Operation not permitted

And finally I listed the file preventing docker container to be removed.


user@server:~$ sudo ls -l /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev
total 0
---------- 1 root root 0 окт 31 18:17 random
---------- 1 root root 0 окт 31 18:17 urandom

Bingo! No permissions to access to file. Even for superuser! It is definitely extended attributes.

user@server:~$ sudo lsattr /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom
----i----------------- /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom

Here it is. Nice. After changing immutable bits we'll be able to retry removing container itself or any of its files. Finally

user@server:~$ sudo chattr -i /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/random
user@server:~$ docker ps -a|grep ‘Dead|Removal’|sed -e 's/\s.*$//'|xargs docker rm
896c259600c9

vou a la.

Docker - Cannot remove dead container

24 Answers24

Linked