ZFS list vs ZFS Pool

Question

I'm software engineer, recently I built my Linux box and wanted to explore more sys-admin type of tasks. I have explored and read a lot about ZFS but I'm getting more confused as each post describe it differently.

Everything works fine, but I don't know how ZFS Engine calculates spaces and similar command make me confused.

$ df -h

Filesystem                                      Size  Used Avail Use% Mounted on
devtmpfs                                         16G     0   16G   0% /dev
tmpfs                                            16G  8.0K   16G   1% /dev/shm
tmpfs                                            16G  122M   16G   1% /run
tmpfs                                            16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/centos_b4--2e--99--49--d4--6d-root   50G  7.8G   43G  16% /
/dev/nvme0n1p2                                 1014M  189M  826M  19% /boot
/dev/nvme0n1p1                                  200M   12M  189M   6% /boot/efi
/dev/mapper/centos_b4--2e--99--49--d4--6d-home  399G  122M  399G   1% /home
mypool                                          9.5T  256K  9.5T   1% /mypool
mypool/data                                     11T   833G  9.5T   8% /mypool/data
tmpfs                                           3.2G     0  3.2G   0% /run/user/1000

$ zpool list

NAME                SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
mypool              14.5T 1.12T  13.4T        -         -     0%     7%  1.00x    ONLINE  -

$ zfs list

NAME                     USED  AVAIL     REFER  MOUNTPOINT
mypool        834G  9.42T      145K  /mypool
mypool/data   834G  9.42T      834G  /mypool/data

$ zpool status mypool

 pool: mypool
 state: ONLINE
  scan: none requested
config:

        NAME               STATE     READ WRITE CKSUM
        mypool  ONLINE       0     0     0
          raidz1-0         ONLINE       0     0     0
            sda            ONLINE       0     0     0
            sdb            ONLINE       0     0     0
            sdc            ONLINE       0     0     0
            sdd            ONLINE       0     0     0

What's make it more confusing is different answer from different ZFS Storage calculators:

Answer 1

the ZPOOL is the big bubble, and then the ZFS can be multiple inside bubbles, or in your case, can be just one bubble

/usr/sbin/zfs list is just reporting the sum of all the zfs when it lists mypool (e.g. if you had a second zfs, mypool/data2, and it happened to be .6TB USED, your mypool line would show 10TB USED, and mypool/data would still be 9.4TB USED); and, your /usr/sbin/zpool list would unafected and still showing 14.5T SIZE

the inner bubble doesn't always have to be the same size as the outer; zfs will grow as it can, unless you turn on quotas; i.e. the ability to cap those inner bubbles; and note, you can move or remove that cap also)

/usr/sbin/zfs get quota,logicalused mypool/data
sudo zfs set quota=10T mypool/data
sudo zfs set quota=12T mypool/data
sudo zfs set quota=10T mypool/data
/usr/sbin/zfs get quota,logicalused mypool/data
sudo zfs set quota=none mypool/data
/usr/sbin/zfs get quota,logicalused mypool/data

Answer 2

For the difference between zpool list and zfs list, see this answer. Now, on to your specific pool and the capacity numbers you’re seeing.

First, what is the difference between the logical and physical view of space usage? The physical view (zpool list) is simplest — it tells you how many bytes are currently being stored on disk. However, this is only rarely the number you actually want, because it doesn’t take into account overheads and doesn’t tell you where the data usage is coming from, which the logical view (zfs list) does know about. Let’s break those down one at a time:

• Overheads: when you create a RAIDZ1 pool, it approximately means that one of the 4 disks in the pool is reserved for storing parity data, so that if a disk dies you can still use the pool / swap in a new disk and fix the pool. This is why the calculator is telling you that 12TiB will be available in the pool (only 3 of the 4 disks can actually store data; 3*4TiB = 12TiB). It’s also why the total capacity used / available reported by zpool list is about 4/3 larger than what zfs list reports. There are also other overheads as well, like ZFS storing filesystem metadata, fragmentation making small slices of a RAID stripe un-allocatable, etc., and zfs list is aware of these and subtracts them out of the “available” number for you.
• Where usage is coming from: in your case this is still pretty easy: all the storage usage for mypool/data is being counted towards the usage for its parent mypool as well. This is useful if you want to limit the storage used for all filesystems under a parent, because you can put a quota on that parent filesystem and the sum of all child filesystems must remain lower, although notably this will also reduce the AVAIL number you see (as will creating a reservation, the opposite of a quota). This can also get trickier when you start creating snapshots and clones on your system because multiple filesystems / snapshots can be using the same data at once. The two main views of usage data with snapshots are “what data is referenced by my snapshot?” and “what data is uniquely referenced by my snapshot?” I encourage you to read the man page for zfs to get more details about the many ways that ZFS can report capacity usage.

Finally, ignore the df -h output — it is just showing you repeats of stuff that ZFS can report to you, and I can never remember whether it’s telling you about storage use in the logical or physical layer. In your example it looks like it tells you logical total space available and logical referenced data, but like I said above, these values get tricky if you start using more advanced features, so it’s better just to use zfs list for almost all purposes so it’s clearer how to interpret a given value.