Docker Machine: No space left on device

Question

I'm trying to set up Docker Machine with Docker Compose.

Scenario 1 (without Docker Machine)
If I run docker-compose up -d without Docker Machine, it creates my 3 linked containers as intented (nginx+mongodb+nodejs).

Scenario 2 (with Docker Machine)
Then I create a VM using Docker Machine and tell Docker to talk to that machine with eval $(docker-machine env streambacker-dev).

At this point, if I ssh to my docker machine and run df -h, I get:

docker machine df -h

If I then run docker-compose up -d, I get a "no space left on device" error while downloading the last container.

"tmpfs" seems to be indeed a bit full after that:

docker machine df -h

Checking the --virtualbox-disk-size option shows that it defaults to 20000 MB, which I think is what we can see as "/dev/sda1" on both pictures. So why are containers filling up "tmpfs" n and what exactly is "tmpfs"? Is is a temporary download directory? How can I create more space for my containers?

Thanks!

For information, I'm using Docker Machine 0.4.0-rc2 and Docker Compose 1.3.2.

The [`tmpfs`](https://en.wikipedia.org/wiki/Tmpfs) has nothing to do with `--virtualbox-disk-size`. It is a filesystem (like a RAM disk) mounted in memory and nothing there is accessing your disk. — Henrik Sachse, Aug 10 '15 at 07:43
check with the dangling stuff of the docker, like dangling images, dangling volumes and delete them. `docker rmi $(docker images -f dangling=true -q)` and `docker volume rm $(docker volume ls -f dangling=true -q)` — Jinna Balu, Jan 08 '19 at 06:36

score 163 · Answer 1 · edited Jun 22 '18 at 12:30

163

I had the same error ([ERROR] InnoDB: Error number 28 means 'No space left on device') and solve it this way:

1 . Delete the orphaned volumes in Docker, you can use the built-in docker volume command. The built-in command also deletes any directory in /var/lib/docker/volumes that is not a volume so make sure you didn't put anything in there you want to save.

Warning be very careful with this if you have some data you want to keep

Cleanup:

$ docker volume rm $(docker volume ls -qf dangling=true)

Additional commands:

List dangling volumes:

$ docker volume ls -qf dangling=true

List all volumes:

$ docker volume ls

2 . Also consider removing all the unused Images.

First get rid of the <none> images (those are sometimes generated while building an image and if for any reason the image building was interrupted, they stay there).

here's a nice script I use to remove them

docker rmi $(docker images | grep "^<none>" | awk '{print $3}')

Then if you are using Docker Compose to build Images locally for every project. You will end up with a lot of images usually named like your folder (example if your project folder named Hello, you will find images name Hello_blablabla). so also consider removing all these images

you can edit the above script to remove them or remove them manually with

docker rmi {image-name}

edited Jun 22 '18 at 12:30

Bamieh

10,358
4
31
52

answered May 19 '16 at 16:49

Mahmoud Zalt

30,478
7
87
83

3

On Mac I had to use single quotes ( ' ) around the argument values: `docker rmi $(docker images | grep '^' | awk '{print $3}')`. Removing stopped containers can be done with `docker rm $(docker ps -qa --no-trunc --filter "status=exited")` as described here: http://stackoverflow.com/a/32723127/619659. – Webdevotion May 25 '16 at 13:38
1

I had so many unused images, I reached my shell's character limit. That's why I did `docker images | grep "^" | xargs docker rmi`. – Matthias Braun Dec 30 '16 at 14:19
1

Removing unneeded images helped for me. – akauppi Dec 30 '16 at 15:44
1

`docker volume prune` – justin.m.chase Mar 04 '19 at 21:09
.. see my step-by-step description for resizing **/var/lib/docker** using **resize2fs** in another thread: https://stackoverflow.com/questions/32485723/docker-increase-disk-space#57711199 – Alex Aug 29 '19 at 13:38
It helped me so i believe that indeed some capacity has been reached - so question: What is this liimtation? Can it be configured / monitored somehow? – JJ Roman Jan 08 '21 at 20:58

wtekiela · Accepted Answer · 2015-11-24T14:56:57.810

Like said above, the tmpfs has nothing to do with --virtualbox-disk-size. It seems like boot2docker mounts tmpfs into memory, so you need to dedicate more memory to your virtualbox vm. You can do it by specifying the --virtualbox-memory parameter.

   --virtualbox-memory "1024"
Size of memory for host in MB [$VIRTUALBOX_MEMORY_SIZE]

Defaults:

$ docker-machine create --driver virtualbox testA
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testA
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testA:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    112.7M    783.9M  13% /

With --virtualbox-memory set to 8096

$ docker-machine create --driver virtualbox --virtualbox-memory 8096 testB
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testB
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testB:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     6.9G    112.4M      6.8G   2% /

Not intuitive indeed! However, this seems to be the problem with this and fixed it for me. So, +1. — Ray Pendergraph, Nov 13 '15 at 16:03
Fixed the problem for me too, I don't understand how but, thank you! — Bendihossan, Nov 30 '15 at 16:05
Any way to increase the default size so that I don't have to specify it manually each time? — stefanbschneider, Aug 07 '19 at 12:49

Alex Efimov · Answer 3 · 2021-08-10T14:12:43.060

89

If you are using Docker Community Edition:

 docker system prune --volumes

If you are using boot2docker (docker-machine) clear the volumes that are orphaned:

 docker volume rm $(docker volume ls -qf dangling=true)

Clear unused images:

 docker rmi $(docker images -q -f "dangling=true")

edited Aug 10 '21 at 14:12

answered Feb 21 '17 at 15:08

Alex Efimov

3,335
1
24
29

9

`docker system prune` helped me – Maxim Yefremov Aug 21 '18 at 14:16
2

`docker system prune` ftw – Michael Guild Sep 18 '18 at 04:45
1

`docker volume prune` for just volumes, system prune will not clear the dangling volumes. – justin.m.chase Mar 04 '19 at 21:09
When the above doesn't clear you enough space, here's the nuclear option: `docker rmi $(docker images -q)`. Add `--force` if you must when it skips a bunch of layers that are included in multiple containers. – EmptyArsenal Apr 30 '20 at 15:18
You don't need two commands for volumes and the rest: `docker system prune --volumes` – jannis Aug 09 '21 at 22:20

score 15 · Answer 4 · edited Jun 20 '20 at 09:12

A. REMOVE UNUSED IMAGES

using the docker rm or docker rmi commands you can remove the images that you don't need. Actually exist an image that helps in this task (martin/docker-cleanup-volumes). The basis is to start selectig from your images and containers list:

docker ps -a -s

B. MODIFY THE DOCKER JSON DESCRIPTOR

it's mentioned in some forums. The idea is to increment the descriptor located in ~/.docker/machine/machines/default/config.json . The param seems to be DiskSize but i don't know if it works in other OSs (not in windows).

C. LINUX RESIZE:

in Windows OS, docker machine or boot2docker is in fact a virtualbox vm, then you can follow the procedure to resize the disk. Take care to backup the files. The general procedure is to make a resize in virtualbox and then use an utilitary called gpartd to modify the space perceived by linux in its partitions. There are some links to do this procedure referenced below:

D. RECREATE THE DOCKER-MACHINE / BOOT2DOCKER

The idea is recreate the default docker-machine. The following commands can illustrate you. Note that as you are re-creating the boot2docker, you will lost the previous downloaded docker images.

docker-machine rm default

docker-machine create --driver virtualbox --virtualbox-disk-size "100100" default

docker-machine env default

then you can go to virtual box and see the boot2docker space with the command "df -h"

Thanks. I was working on Windows. I made it by recreating the docker-machine. — KinoP, Jun 20 '16 at 02:47
Thank you, recreating the docker-machine worked for me (I'm on windows) — Lucha Laura Hardie, Sep 27 '17 at 15:26

kwabena Adu-Darkwa · Answer 5 · 2020-10-20T05:13:35.850

3

Run docker system df to see what is taking up space on your docker

Then run:

docker system prune --all --force

to remove all hidden and unused containers.

docker system prune does not remove all unused containers.

edited Oct 20 '20 at 05:13

answered Oct 19 '20 at 06:16

kwabena Adu-Darkwa

31
3

That's the answer! I was looking for that "docker system df" right there ;) – Brandt Jun 30 '23 at 15:10

score 2 · Answer 6 · answered Sep 06 '17 at 22:41

2

On docker osx / I was able to press a button [Move Disk Image] and it successfully moved the Docker.qcow2 (presumably containing containers / images)

initially - when machines started - I was still getting a No space left on device error but it resolved shortly after.

answered Sep 06 '17 at 22:41

johndpope

5,035
2
41
43

where did you move it to that was a better location (allowed more space)? – littleforest Oct 07 '20 at 22:11
external ssd drive – johndpope Oct 21 '20 at 02:59

aabiro · Answer 7 · 2018-10-01T08:28:30.480

I ran into this problem and could not add additional space with the docker UI for mac, I installed docker with homebrew and ran the following command when creating my machine:

docker-machine create --driver virtualbox --virtualbox-memory "2048" --virtualbox-disk-size "40000" default

this adds double the space for memory and disk size to the virtualbox that I had before and you can add the settings size here that you need as you see fit

score 0 · Answer 8 · answered Aug 20 '21 at 07:42

Open file /lib/systemd/system/docker.service with your favorite text editor and replace the following line where /new/path/docker is a location of your new chosen docker directory:

FROM:
ExecStart=/usr/bin/docker daemon -H fd://
TO:
ExecStart=/usr/bin/docker daemon -g /new/path/docker -H fd://

When ready stop docker service:

# systemctl stop docker

check that docker service is stopped, the following command should yield no result

# ps aux | grep -i docker | grep -v grep

reload daemon

# systemctl daemon-reload

Once this is done create a new directory you specified above and optionally rsync current docker data to a new directory:

# mkdir /new/path/docker
# rsync -aqxP /var/lib/docker/ /new/path/docker

# systemctl start docker

confirms that docker runs into new directory

#  ps aux | grep -i docker | grep -v grep

https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux

Docker Machine: No space left on device

8 Answers8

Linked