1

I am using a persistent volume. While mounted in a container, I got an error message that there was no space left on the device. On inspection, the volume size was 256GB. However, there is much more room on the host.

Some further details: The docker storage driver is overlay2. df reveals that there is a filesystem /dev/mapper/vg1-docker that is 256GB. During a download (over the internet) this filesystem became 100% full and docker did not automatically resize the filesystem to make more space, even though there was much more space available on the host.

The host is native linux (ubuntu). Here is the output of df -i:

Filesystem               Inodes  IUsed    IFree IUse% Mounted on
udev                   24754746    429 24754317    1% /dev
tmpfs                  24757974    762 24757212    1% /run
/dev/xvda2              6537216 142008  6395208    3% /
tmpfs                  24757974      1 24757973    1% /dev/shm
tmpfs                  24757974      3 24757971    1% /run/lock
tmpfs                  24757974     18 24757956    1% /sys/fs/cgroup
/dev/mapper/vg1-docker 16384000 985506 15398494    7% /var/lib/docker
/dev/xvda1                65536    313    65223    1% /boot
tmpfs                  24757974     10 24757964    1% /run/user/1001

I can't find any information on increasing the size of a docker volume.

Note this is not a duplicate of Why has Docker volume run out of space?.

James Hirschorn
  • 7,032
  • 5
  • 45
  • 53
  • 1
    Can you edit the question to include the actual commands you're running and the actual error you're getting? Potentially relevant as well is the line that says `Storage Driver:` in the output of `docker info` (on most modern systems it will say `overlay2` but if it says `devicemapper` there's potential issues). – David Maze Nov 09 '19 at 10:25
  • We can also use the output of `df -i` and whether you are running on a native Linux host or a desktop (Windows/Mac) environment. – BMitch Nov 09 '19 at 13:20
  • @DavidMaze I have added the information you asked about. It was doing a download from the internet, so presumably the exact command is not important. – James Hirschorn Nov 10 '19 at 16:59
  • @BMitch I added the information you suggested. – James Hirschorn Nov 10 '19 at 17:06

2 Answers2

1

df reveals that there is a filesystem /dev/mapper/vg1-docker that is 256GB. During a download (over the internet) this filesystem became 100% full and docker did not automatically resize the filesystem to make more space, even though there was much more space available on the host.

I believe you have answered the question yourself. Docker ran out of disk space because the partition you isolated it onto ran out of disk space. Docker doesn't, nor should it, automatically resize disk space. That's a task for the sysadmin, not the container engine. The purpose of creating a separate partition for docker is often to ensure that docker cannot take up all of the disk space on the root filesystem, which could bring down the entire host.

BMitch
  • 231,797
  • 42
  • 475
  • 450
  • I assumed that docker had automatically created the partition `/dev/mapper/vg1-docker` for me. You seem to be saving that docker does not do that. So there is a way to isolate docker volumes to some partition? I guess it is possible I did this several months ago when I setup the docker and then forgot about it. But I don't even remember how to isolate docker to a specific partition. – James Hirschorn Nov 11 '19 at 02:15
  • @JamesHirschorn By default, docker stores data in `/var/lib/docker`. It appears that you've created a partition there using LVM in Linux. This is a good best practice to limit Docker's resources, and you've hit that limit. – BMitch Nov 11 '19 at 03:07
  • Yes, that must be it. – James Hirschorn Nov 11 '19 at 04:19
0

You can check current volumes list on your machine by this command

docker volume ls

In your case, I think you have too many docker containers run on your machine over time (Your docker volumes used total should not over 10% of disk space). The easiest way, just prune unnecessary volume which out of date by this command, to take back disk space

docker volume prune -f

If you need to redeem more space. Just prune unused images and networks also.

Truong Dang
  • 3,119
  • 1
  • 15
  • 21
  • Why do you say "Your docker volumes should not over 10% of disk space"? Btw, I did `docker volume prune` (before reading your answer) and lost my volume. I will have to recreate it and try from the beginning again. – James Hirschorn Nov 09 '19 at 06:19
  • 1
    Just my experience, each time I meet this problem. Total volumes in my server around 10% of disk space – Truong Dang Nov 09 '19 at 06:24
  • No. docker volume prune just remove unused volume (from exited containers). If you container current running and use those volumes. It does not remove at all – Truong Dang Nov 09 '19 at 06:35
  • right, but I had stopped my container before I did the prune command. – James Hirschorn Nov 09 '19 at 06:42
  • `docker volume prune -f` gives: `Total reclaimed space: 0B`. If docker volumes can't use the host disk space effectively, there should be some notice in the documentation. – James Hirschorn Nov 10 '19 at 20:04