2

I just installed docker version 17.06.2-ee-3, build 915cbaa on Red Hat version 7.4. When I try and build an image it fails with No space left on device even thought there is plenty of space left. I found this post: https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/ which may not work any more, but I followed it anyway and now docker info shows:

 Data Space Used: 30.68GB
 Data Space Total: 268.4GB
 Data Space Available: 134.7GB
 Metadata Space Used: 32.23MB
 Metadata Space Total: 2.147GB
 Metadata Space Available: 2.115GB

I can successfully build this same image on with Red Had 6.7 with docker version 1.7.1, build 786b29d and on Red Hat 7.2 with docker version version 1.11.1, build 5604cbe.

As requested:

$ df -Thi
Filesystem                      Type     Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rhel-root           xfs         87M  180K   87M    1% /
devtmpfs                        devtmpfs    16M  1.3K   16M    1% /dev
tmpfs                           tmpfs       16M     1   16M    1% /dev/shm
tmpfs                           tmpfs       16M  1.4K   16M    1% /run
tmpfs                           tmpfs       16M    16   16M    1% /sys/fs/cgroup
/dev/sda3                       xfs        250K   337  250K    1% /boot
/dev/mapper/vg_lvm1-elucid_vol1 ext4        47M    11   47M    1% /elucid
/dev/mapper/rhel-home           xfs         97M   16K   97M    1% /home
192.168.10.5:/projects          nfs4       625M  6.4M  619M    2% /projects
192.168.10.5:/home              nfs4       3.8M  174K  3.6M    5% /home_bekku
tmpfs                           tmpfs       16M    10   16M    1% /run/user/42
tmpfs                           tmpfs       16M     1   16M    1% /run/user/1000
tmpfs                           tmpfs       16M     1   16M    1% /run/user/522

$ df -H /var/lib/docker/
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root  187G   52G  135G  28% /

$ du -sH .
7990180 .

In response to Tarun Lalwani's request to run journalctl -f -n10 here is the output of that around the time I get the out of space message:

Sep 28 08:11:38 liszt NetworkManager[2449]: <info>  [1506600698.9768] device (veth1cb0d8c): link connected
Sep 28 08:11:38 liszt NetworkManager[2449]: <info>  [1506600698.9770] device (docker0): link connected
Sep 28 08:11:39 liszt kernel: docker0: port 1(veth1cb0d8c) entered disabled state
Sep 28 08:11:39 liszt kernel: docker0: port 1(veth1cb0d8c) entered disabled state
Sep 28 08:11:39 liszt avahi-daemon[2350]: Withdrawing workstation service for vethb5fa5c3.
Sep 28 08:11:39 liszt kernel: device veth1cb0d8c left promiscuous mode
Sep 28 08:11:39 liszt kernel: docker0: port 1(veth1cb0d8c) entered disabled state
Sep 28 08:11:39 liszt avahi-daemon[2350]: Withdrawing workstation service for veth1cb0d8c.
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.606+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt NetworkManager[2449]: <info>  [1506600699.6083] manager: (vethb5fa5c3): new Veth device (/org/freedesktop/NetworkManager/Devices/50)
Sep 28 08:11:39 liszt NetworkManager[2449]: <info>  [1506600699.6110] device (veth1cb0d8c): released from master device docker0
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.611+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.613+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.615+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.617+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.619+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.621+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt libvirtd[3357]: 2017-09-28 12:11:39.623+0000: 3357: error : virNetDevSendEthtoolIoctl:2939 : ethtool ioctl error: No such device
Sep 28 08:11:39 liszt kernel: XFS (dm-5): Unmounting Filesystem

Anyone know how I get around this issue and actually use docker here?

Larry Martell
  • 3,526
  • 6
  • 40
  • 76
  • Maybe there's a problem with the inodes? Could you run: df -Thi – Sergiu Sep 27 '17 at 21:04
  • Also you might be able to visit: https://unix.stackexchange.com/questions/203168/docker-says-no-space-left-on-device-but-system-has-plenty-of-space as it seems like the same issue you are having – Sergiu Sep 27 '17 at 21:06
  • Yes, I saw that. That is from a older version of docker, and the link to the docker docs is broken. The other link is to the page I referenced in my original post, and it says there 'At some point, Docker storage driver internals have changed significantly, and the technique described here doesn’t work anymore.' – Larry Martell Sep 27 '17 at 21:22
  • What does `df -H /var/lib/docker/.` show and in your build directory, what does `du -sH .` show? – BMitch Sep 27 '17 at 21:35
  • [Don't use a loopback file on RHEL](https://stackoverflow.com/questions/37672018/clean-docker-environment-devicemapper/37681340#37681340), create an LVM thinpool. – Matt Sep 27 '17 at 22:36
  • or use overlay2, it's not prod supported in Docker EE but doesn't break as much as a loopback devicemapper file will. – Matt Sep 27 '17 at 22:40
  • Try running `journalctl -f -n10` in another terminal and start the build. see if you find anything interesting in the logs? – Tarun Lalwani Sep 28 '17 at 06:42

1 Answers1

4

I finally solved this. Using watch df -h while I was building the image I saw that docker mounted a 10G partition which filled up. I could not find out how to increase that, and I spoke with someone at docker who told to me put these settings in the /etc/docker/daemon.json file:

{   
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.basesize=20G"
    ]
}

After I did that and restarted docker the mounted partition was 20G and I could build my image.

Larry Martell
  • 3,526
  • 6
  • 40
  • 76
  • Adding those lines breaks my engine, it fails to load the config. The commandline for configs was not helping me either. –  Apr 23 '20 at 21:01