3

I don't know actually if this is more a "classic" linux or a docker question but:

On an VM where some of my docker containers are running I've a strange thing. /var/lib/docker is an own partitionwith 20GB. When I look over the partition with df -h I see this:

eti-gwl1v-dockerapp1 root# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  815M  7.0G  11% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda2        12G  3.2G  8.0G  29% /
/dev/sda7       3.9G   17M  3.7G   1% /tmp
/dev/sda5       7.8G  6.8G  649M  92% /var
/dev/sdb2        20G   47M   19G   1% /usr2
/dev/sdb1        20G  2.9G   16G  16% /var/lib/docker

So usage is at 16%. But when I now navigate to /var/lib and do a du -sch docker I see this:

eti-gwl1v-dockerapp1 root# cd /var/lib
eti-gwl1v-dockerapp1 root# du -sch docker
19G     docker
19G     total
eti-gwl1v-dockerapp1 root#

So same directory/partition but two sizes? How is that going?

gurbelunder
  • 165
  • 4
  • 12
  • 1
    What filesystem you are using? – KamilCuk Dec 09 '20 at 11:55
  • /var/lib/docker itself is ext4, my docker container are using overlay2 – gurbelunder Dec 09 '20 at 12:38
  • `overlay2` - how do you think "overlay" works? If you count sizes of all files, if you have `dir1` and mount `dir2` as an overlay, how do you think the size will change? How do you think the size on the disc will change? – KamilCuk Dec 09 '20 at 13:10

3 Answers3

2

This is really a question for unix.stackexchange.com, but there is filesystem overhead that makes the partition larger than the total size of the individual files within it.

Vercingatorix
  • 1,838
  • 1
  • 13
  • 22
1

du and df show you two different metrics:

  • du shows you the (estimated) file space usage, i.e. the sum of all file sizes
  • df shows you the disk space usage, i.e. how much space on the disk is actually used

These are distinct values and can often diverge:

  • disk usage may be bigger than the mere sum of file sizes due to additional meta data: e.g. the disk usage of 1000 empty files (file size = 0) is >0 since their file names and permissions need to be stored
  • the space used by one or multiple files may be smaller than their reported file size due to:
    • holes in the file - block consisting of only null bytes are not actually written to disk, see sparse files
    • automatic file system compression
    • deduplication through hard links or copy-on-write

Since docker uses the image layers as a means of deduplication the latter is most probably the cause of your observation - i.e. the sum of the files is much bigger because most of them are shared/deduplicated through hard links.

acran
  • 7,070
  • 1
  • 18
  • 35
1

du estimates filesystem usage through summing the size of all files in it. This does not deal well with the usage of overlay2: there will be many directories which contain the same files as contained in another, but overlaid with additional layers using overlay2. As such, du will show a very inflated number.

I have not tested this since my Docker daemon is not using overlay2, but using du -x to avoid going into overlays could give the right amount. However, this wouldn't work for other Docker drivers, like btrfs, for example.

Leonardo Dagnino
  • 2,914
  • 7
  • 28