0

We have an Ubuntu server that runs a few test environments under Docker. The system ran out of space today, and I eventually found that /var/lib/docker had grown to 826 GB with /var/lib/docker/volumes alone making 600 GB thereof.

Turns out that docker has created over 1,300 volumes:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          12        12        9.271GB   447.8MB (4%)
Containers      12        10        405.9MB   62.84MB (15%)
Local Volumes   1307      8         634.8GB   630.8GB (99%)
Build Cache     3586      0         19.05GB   19.05GB

I tried docker volume prune (and also with force-pruning images and containers before that), but that removed only a handful volumes.

We have another server with the same test environments under Docker, and it's looks totally different (i.e. normal) with only 9 volumes:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          13        13        10.07GB   447.8MB (4%)
Containers      14        10        312.2MB   111.7MB (35%)
Local Volumes   9         9         5.212GB   0B (0%)
Build Cache     0         0         0B        0B

If I check one of these volumes to find out which contain er or image it belongs to, I get nothing:

$ docker ps -a --filter volume=ff17641242297f3f9f3a63b2c328168f93c514e6e41cc3ed19bfbcb192acb5c7
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

The I tried a more radical solution:

$ docker system prune -a -f

It deleted a lot of build cache objects (21 GB in total), but no volumes.

How can it be that Docker has created 1,300 volumes that cannot be pruned? And what can I do to fix that?

Note that I found Why did my docker volume run out of disk space?, but the solutions given in the answers there do not work, so I assume my problem is a different one.

not2savvy
  • 2,902
  • 3
  • 22
  • 37
  • 1
    This seems to be more of an administration question than a programming question. Do you have a [mcve] or any code that shows what might be creating the volumes? – David Maze Jul 17 '23 at 15:02
  • It now seems to be a problem within docker itself. I was just creating images and containers regularly over months, so hard to tell a specific code. – not2savvy Jul 18 '23 at 07:29
  • Have you tried `docker volume prune -a`. Without a [mcve], this is just a random guess. – BMitch Aug 06 '23 at 11:53
  • @BMitch Yes, I did. I’m afraid I don’t know how to reproduce the issue, so can’t provide an example. Maybe this question should be relocated to superuser? – not2savvy Aug 07 '23 at 08:51

1 Answers1

0

This command eventually did it:

docker volume rm $(docker volume ls -q --filter dangling=true)

According to the docs, docker volume prune -f should actually do the job, but it didn't. According to this answer, the above is "the old way" to do it, but obviously, it is the way it actually works, too.

not2savvy
  • 2,902
  • 3
  • 22
  • 37