2

VMware 6.5 announced to support UNMAP commands (Space Reclamation from thin LUNs on storage systems) with VMFS6 and Linux virtual machines which supports SCSI Primary Commands - 4 (SPC-4). Which version of Linux Kernel start to support SPC-4 for VMware 6.5 UNMAP? This is helpful article.

BBK
  • 142
  • 8

2 Answers2

3

It's less of a question of "When will Linux support SPC-4 for UNMAP" (you can always try and send raw SCSI commands and Linux doesn't have to actually "understand" those) so I think you meant to ask "which version of Linux automatically advertises discard on appropriate ESXi disks?".

For the past few years so long as the "disk" says it can support the SCSI standard SPC-2 or higher Linux will go on to check and expose discard (aka TRIM or UNMAP) support if available. Exposing discard support went in with commit https://github.com/torvalds/linux/commit/c98a0eb0e90d1caa8a92913cd45462102cbd5eaf which eventually turned up in 2.6.39. See https://github.com/torvalds/linux/blob/v4.8/include/scsi/scsi_device.h#L546 which shows how reading the VPD pages only happens if a) It is somehow forced (via explicit quirks for er, "quirky" devices) or b) the device claims to implement at least the SCSI SPC-2 specifcation. In turn being able to read VPD pages is needed for checking if the device supports thin provisioning over on https://github.com/torvalds/linux/blob/v4.8/drivers/scsi/sd.c#L2840 .

I've checked a thin provisioned VMDK disk on ESXi 6.0, version 11 VM running Ubuntu 16.04, Guest OS set to Ubuntu 64 bit. If you look at the "Logical block provisioning" VPD page (via sg_vpd -p lbpv) for the disk you are told:

Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 1
  Write same (16) with unmap bit supported (LBWS): 0
  Write same (10) with unmap bit supported (LBWS10): 0

(I had to set the advanced option EnableBlockDelete to 1 on the VM to get this)

If you look at the READCAPACITY(16) results (via sg_readcap -16):

Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=1, lbprz=1

So again it's claiming to be thin.

However sg_inq shows the disk only claims to support SCSI-2:

standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  LU_CONG=0  version=0x02  [SCSI-2]

(SCSI-2 is a few revision below SPC-2 see https://github.com/torvalds/linux/blob/v4.8/include/scsi/scsi.h#L253 for how Linux orders SCSI spec versions).

So Linux won't advertise discard:

grep . /sys/block/sdc/queue/discard_max_bytes
0

and anything that tries to use discard will fail for me:

# blkdiscard --offset 0 --length=2048 /dev/sdc
blkdiscard: /dev/sdc: BLKDISCARD ioctl failed: Operation not supported

Despite this it's possible to manually send raw SCSI UNMAPs down:

# sg_unmap --lba=0 --num=2048 /dev/sdc

(note that there's a minimum size of 1MByte for unmap operations or you'll get an error back)

TLDR; Advertising discard in Linux was added in 2.6.39 but nearly any Linux can be made to manually passthrough SCSI UNMAP commands (and on ESXi your VM must meet the correct requirements).

Anon
  • 6,306
  • 2
  • 38
  • 56
  • Hi. **I had to set the advanced option EnableBlockDelete to 1 on the VM to get this**. How you did this? – BBK Nov 28 '16 at 12:28
  • `esxcli --server=server_name system settings advanced set --int-value 1 --option /VMFS6/EnableBlockDelete` ? – BBK Nov 28 '16 at 12:43
  • Is vCenter must have? Do we need vmware tools to be installed in guest OS? – BBK Dec 08 '16 at 10:13
  • Which kernel version start to support "SPC-4"? – BBK Dec 08 '16 at 11:32
  • 1
    @BBK If you are willing to *manually* send raw SCSI UNMAP the kernel version doesn't matter (above I had only *SCSI-2*). Kernels from 2.6.39 onwards will check for and advertise *discard* in most applicable situations (and it is discard that filesystems use to say say a region can be "forgotten") but your device *must* claim to do SPC-2 or above. The vCenter version and whether VMware tools is installed are irrelevant to discard advertising but the VM configuration and ESXi version DO matter. See the "VMware vSphere 6.5" link above for configuration settings you MUST have on ESXi 6.5. – Anon Dec 09 '16 at 07:46
  • 2
    Another way of putting it is: you are probably more concerned about whether Linux advertises discard than whether you can send raw SCSI UNMAP commands. Linux doesn't have to support all of SPC-4 in order to advertise discard but rather VMware chose to fulfil Linux's criteria for discard (disk claims to do at least SPC-2, disk advertises that it is thin, disk advertises commands that make it thin, disk and controller aren't blacklisted for bad behaviour) by making their disks (claim to) implement SPC-4 and changing the version number on their disks and the criteria hasn't changed since 2.6.39. – Anon Dec 09 '16 at 08:07
1

UNMAP support appears at the earliest in the 2.6.27 to 2.6.28 timeframe for the ext4 filesystem. The Wikipedia page for TRIM/UNMAP lists kernel version 2.6.28-25 specifically.

UNMAP support is tied specifically to the filesystem, because it's the filesystem that knows when it's done with a range of blocks, and can release them back to the underlying device. Make sure to choose a filesystem that supports it!

There's a good summary under the Community Driven Feature: Discard Support section of this SNIA presentation. As described, kernel versions from 2.6.32 should have good support. In particular, the changes made it into mainstream RHEL6. So, by this point, it's been out there a while!

Mike Andrews
  • 3,045
  • 18
  • 28