3

i have problem with docker overlay2 folder.

When I enter du -sj /var/lib/docker/overlay2 in the time of writing it says:

85G overlay2/

And it keeps growing up. I tried to prune system with command like:

docker system prune -a

But output is :

Total reclaimed space: 0B

I also tried to add max-file and max-size to my docker compose. After adding these lines I recreated containers. But problem is still there, can it be solved in any way, just to stop eating disk space ?

EDIT:

docker info:

Server:
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 6
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem:
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local \logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-91-generic
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 62.86GiB
Name: machine
ID: TRX2:AJZG:LNJF:UCPW:MQQB:2PLR:R7KM:VHEV:KBOG:IBT4:JX4R:WR6V
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/\ Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false\

docker image: (for the security reasons i wont show the names)

TAG IMAGE ID CREATED SIZE
latest 667ac029b200 2 years ago 7.6GB
latest cb9df5aac4ac 2 years ago 12.9GB
latest d445c0adc9a5 3 weeks ago 220MB
latest 227d9f0554d3 2 years ago 1.34GB
latest b80092729008 2 years ago 758MB
latest f32a97de94e1 2 years ago 25.8MB
Nikola Roganovic
  • 43
  • 1
  • 6
  • What containers are you running? How much space do they take up (the answer to that is likely a little over 85G)? Include the output of `docker info`, `docker image ls -a`, and `docker container ls -a` in your question. – BMitch Jan 06 '22 at 11:49

3 Answers3

3
#!/bin/bash
# Remove exited containers
/usr/bin/docker ps -a -q -f status=exited | xargs --no-run-if-empty docker rm -v

# Remove dangling images
/usr/bin/docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi

# Remove unused images
/usr/bin/docker images | awk '/ago/  { print $3}' | xargs --no-run-if-empty docker rmi

# Remove dangling volumes
/usr/bin/docker volume ls -qf dangling=true | xargs --no-run-if-empty docker volume rm

Run with cron. In my case every hour

enter image description here

Facty
  • 472
  • 4
  • 12
1

You have 10 containers running, and the 2 year old images are a strong indication that you're likely mutating the filesystem inside those containers (writing logs, temp files, maybe even installing apps). A prune will not delete those containers, their images, the filesystem changes within those containers, or the logs of the containers. The part that you'll see in overlay2 are those filesystem changes, so use the following to see what files have been created/changed within each container:

docker container diff ${container_name_or_id}

e.g.:

$ docker container diff d7c
C /run
A /run/user
A /run/user/1000
C /tmp
A /tmp/.X11-unix
C /home
A /home/user

Your list will likely be longer, and you can go into the container to delete any files created that you no longer need (using docker exec assuming the container has a shell and other CLI tools).

For more on cleaning up a large overlay2 directory, see is it safe to clean docker/overlay2.

BMitch
  • 231,797
  • 42
  • 475
  • 450
  • So to avoid overlay2 taking up the entire disk (assuming frequent re-deployments), one should make sure, that one never edits/adds files that are part of the image? So `docker container diff ` should always be an empty list? – nitzel Feb 27 '23 at 12:20
  • @nitzel when the image gets rebuilt (e.g. for a new version) and redeployed, the container is recreated and the old container, along with all filesystem changes, can be safely pruned. Old images and long running containers are a code smell in the container ecosystem. Some apps always write to the filesystem, so it's often not possible to get the diff completely empty, but it should be ephemeral. – BMitch Feb 27 '23 at 15:37
0
  1. Check with docker volume ls , it will show volumes and size
  2. whatever not required clear them
  3. If you are using linux based images you can use - apt-get autoremove , to remove orphan packages
Shaktirajsinh Jadeja
  • 414
  • 5
  • 9