Why is Docker filling up /var/lib/docker/overlay2?

Question

Even though I have successfully (?) removed all Docker images and containers, the folder /var/lib/docker/overlay2 still is vast (152 GB). Why? How do I reduce the used disk size?

I have tried to rename the folder (in preparation for a possible removal of the folder) but that caused subsequent pull requests to fail.

To me it appears pretty unbelievable that Docker would need this amount of vast disk space just for later being able to pull an image again. Please enlighten me what is wrong or why it has to be this way.

List of commands run which should show what I have tried and the current status:

$ docker image prune --force
Total reclaimed space: 0B

$ docker system prune --force
Total reclaimed space: 0B

$ docker image prune -a --force
Total reclaimed space: 0B

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

$ docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ du -h --max-depth=1 /var/lib/docker/overlay2 | sort -rh | head -25
152G    /var/lib/docker/overlay2
1.7G    /var/lib/docker/overlay2/ys1nmeu2aewhduj0dfykrnw8m
1.7G    /var/lib/docker/overlay2/ydqchhcaqokdokxzbh6htqa49
1.7G    /var/lib/docker/overlay2/xmffou5nk3zkrldlfllopxcab
1.7G    /var/lib/docker/overlay2/tjz58rjkote2c79veonb3s6qa
1.7G    /var/lib/docker/overlay2/rlnr04hlcudgoh6ujobtsu2ck
1.7G    /var/lib/docker/overlay2/r4ubwsmrorpr08k8o5rko9n98
1.7G    /var/lib/docker/overlay2/q8x21c9enjhpitt365smkmn4e
1.7G    /var/lib/docker/overlay2/ntr973uef37oweqlxr4kmaxps
1.7G    /var/lib/docker/overlay2/mcyasqzo2gry5dvjxoao1opws
1.7G    /var/lib/docker/overlay2/m2k4u58mob6e2db86qqu1e1f8
1.7G    /var/lib/docker/overlay2/lizesless03kch8j7kpk89rcf
1.7G    /var/lib/docker/overlay2/kmu7mjvsopr8o63onbsijb98j
1.7G    /var/lib/docker/overlay2/khgjwqry5drdy0jbwf47gr2lb
1.7G    /var/lib/docker/overlay2/gt70ur50vw3whq265vmpep7ay
1.7G    /var/lib/docker/overlay2/c3tm1fcuekmdreowrfcso7nd4
1.7G    /var/lib/docker/overlay2/7j93t64mt63arj6sewyyejwyo
1.7G    /var/lib/docker/overlay2/3ftxvvg2xg02xuwcb3ut3dq89
1.7G    /var/lib/docker/overlay2/0m3o3lw6b1ggs8m6z4uv6ueqf
1.4G    /var/lib/docker/overlay2/r82rfxme096cq5pg1xz1z5arg
1.4G    /var/lib/docker/overlay2/qric73hv1z3nx4k0zop3fvcm6
1.4G    /var/lib/docker/overlay2/oyb0a5ab5h642y30s6hawj4r9
1.4G    /var/lib/docker/overlay2/oqf9ltfoy36evnkuo8ga2uepl
1.4G    /var/lib/docker/overlay2/ntuwvljxxzqs2oxhgg3enyo7x
1.4G    /var/lib/docker/overlay2/l0oi2lxdrtg42hk2rznknqk0r

$ ls -l /var/lib/docker/overlay2
total 136
drwx------  4 root root     72 Nov 20 13:03 00ep8i7v5bdmhqsxdoikslr19
drwx------  4 root root     72 Feb 28 09:47 026x5e2xns6ui2acym19qfvl7
drwx------  4 root root     72 Apr  2 19:20 032y8d31damevtfymq6yzkyi4
drwx------  4 root root     72 Apr 23 13:42 03wwbyd4uge9u0auk94wwdlig
drwx------  4 root root     72 Jan 15 12:46 04cy91a19owwqu9hyw6vruhzo
drwx------  4 root root     72 Apr  2 14:44 051625a0f856b63ed67a3bc9c19f09fb1c90303b9536791dc88717cb7379ceeb
drwx------  4 root root     72 Dec  3 19:56 059fk19uw70p6fqzei6wnj8s2
drwx------  4 root root     72 Apr 21 15:03 059mddrhqegqhxv1ockejw9gs
drwx------  4 root root     72 Nov 28 11:26 069dwkz92m8fao6whxnj4x9vp
drwx------  4 root root     72 Feb 28 09:47 06h7qo5f70oyzaqgn1elbx5u8
drwx------  4 root root     72 Dec 18 13:27 0756fd640036fa92499cfdcf4bcc3081d9ec16c25eebe5964d5e12d22beb9991
drwx------  4 root root     72 Apr 20 11:32 09rk4gm6x2mcquc5cz0yvbawq
drwx------  4 root root     72 Apr  2 19:55 09scfio3qvtewzgc5bdwgw4f6
drwx------  4 root root     72 May  4 14:00 0ac2a09aa4a038981d37730e56dece4a3d28e80f261b68587c072b4012dc044a
drwx------  4 root root     72 Feb 25 14:19 0c399f5c349ec61ac175525c52533b069a52028354c1055894466e9b5430fbc3
drwx------  4 root root     72 May  4 14:00 0cac39b1382986a2d9a6690985792c04d03192337ea37ee06cb74f2f457b7bb7
drwx------  4 root root     72 Mar  5 08:41 0czco1xx3148slgwf8imdrk33
drwx------  4 root root     72 Apr 21 08:30 0gb2iqev9e7kr587l09u19eff
drwx------  4 root root     72 Feb 20 18:03 0gknqh4pyg46uzi6asskbf8xk
drwx------  4 root root     72 Jan  8 11:43 0gugiou3wqu53os4dageh77ty
drwx------  4 root root     72 Jan  7 11:31 0i8fd5jet6ieajyl2uo1xj2ai
.
.
.

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Do you have running or stopped containers? `docker ps` and `docker ps -a` empty? — Peleion, May 04 '20 at 12:22
Thanks for your interest in my problem! No, no containers are running as shown above with the "docker container ls" command. I ran your suggested ps commands: $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES — Floffy75, May 04 '20 at 14:46

score 11 · Answer 1 · edited May 04 '20 at 18:02

You might have switched storage drivers somewhere along the way, so maybe docker is just cleaning out those drivers but leaving overlay2 as is (I still can't understand why would pulling images would fail).

Let's try this, run docker info and check what is your storage driver:

$ docker info

Containers: 0
Images: 0
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
<output truncated>

If it is not overlay2 (as appears above) try switching to it, and then prune docker images again and check if that cleaned up that folder.

Another possible solution is mentioned in this thread, people are commenting that clearing logs solves this problem, so try the following:

Remove all log files:

find /var/lib/docker/containers/ -type f -name "*.log" -delete

Restart docker daemon (or entire machine):

sudo systemctl restart docker

or

docker-compose down && docker-compose up -d

or

shutdown -r now

I don’t know what’s happening with my machine, it sometimes just errors out for some reason, sometimes `find /var/lib/docker/containers/ -type f -name "*.log" -delete` Solves it, sometimes it just passes to a next error. — mr.xed, Feb 21 '22 at 16:17

Peleion · Answer 2 · 2020-05-04T17:58:21.487

in preparation for a possible removal of the folder

If you are going to delete all data from the Docker directory anyway it is safe to:

Stop Docker Daemon
Remove the /var/lib/docker directory entirely
Restart Docker Daemon

Docker will then recreate any needed data directories.

You can also add:

"log-driver": "json-file",
"log-opts": {"max-size": "20m", "max-file": "3"},

to your /etc/docker/daemon.json to restrict log size and growth in the future or set the log-driver to "journald" to eliminate log files entirely.

score 6 · Answer 3 · answered May 08 '20 at 10:21

Thanks for your input and suggestions!

I believe that I am still using overlay2 as storage driver:

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8
 Storage Driver: overlay2
<output truncated>

I also cleared the logs and restarted the daemon and actually also the entire machine. The problem however remained.

In the end I solved it by stoping the deamon, removing the entire docker folder and restarting the deamon, as suggested above.

df -h
sudo systemctl stop docker
sudo mv /var/lib/docker /var/lib/docker_old
sudo systemctl start docker
sudo rm -rf /var/lib/docker_old
df -h

I fear however that this will not be a permanent solutions and that the problem will come back, but this will hopefully last another year. :)

score 4 · Answer 4 · answered Nov 24 '22 at 20:02

Two things will fill up /var/lib/docker/overlay2:

Docker images: Clean those with docker image prune -a. Note that any images not currently associated with a container will be deleted which requires pulling or building the image again if you needed it.
Container Specific Changes: any write to the container filesystem that isn't going to another mount (like a volume) will cause a copy-on-write that is stored in the container specific layer. You can see these changes with docker diff on a container. Even a metadata change like file ownership, permissions, or a timestamp, can trigger this copy-on-write of the entire file.

Things that are not included in this directory:

Volumes: Named volumes will be stored in /var/lib/docker/volumes by default. You can still prune these with docker volume prune but make sure you have backed up any important data first. A better cleanup is to remove unused anonymous volumes with a command like:
```
docker volume ls -qf dangling=true | egrep '^[a-z0-9]{64}$' | \
  xargs --no-run-if-empty docker volume rm
```
Container Logs: Container logs will be written to /var/lib/docker/containers. If these are taking up space, it's best to have docker automatically rotate those. See this answer for details on rotating logs.

I found this answer helpful to debug where exactly our space issues came from; nowadays a `docker system prune -a` would probably just clean up everything, but it's helpful to see what lead to the situation in the first place, thanks! — Torque, Jun 14 '23 at 07:21

Nick · Answer 5 · 2023-02-13T21:47:12.037

Try to prune everything including volumes (different than the original poster's command):

$ docker system prune --volumes
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all volumes not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N]

That freed up a bunch of space for me and solved my issue. I think the build cache was one of the culprits for me.

souch · Answer 6 · 2023-01-04T09:15:53.603

I had the same problem, /var/lib/docker/overlay2 was using 17 GB even after removing every docker image: docker image rm ...

When you stop a container, it is not automatically removed unless you started it with the --rm flag (prune-containers). Container size can be seen using this command: docker container ls -a -s

I managed to reclaim the space taken by stopped container using this command: docker container prune.

Answer of @Nick is perhaps even better as it cleans every unused docker files: docker system prune --volumes

score 0 · Answer 7 · answered Aug 23 '23 at 20:33

0

Before a hard clean with: docker system prune, try a: docker system df to see disk usage of images, containers and also build cache.

If the build cache size is too large: docker builder prune.

dockerd v20.10.24.

answered Aug 23 '23 at 20:33

Erwann Sant

31
3

Why is Docker filling up /var/lib/docker/overlay2?

7 Answers7