3

I am benchmarking databases above a NVMe SSD. I want to monitor the number of I/O request in the queue in this figure over time to see if the databases fully take advantage of the queues.
enter image description here

I have tried tools like iostat, but the avgqu-sz field is always zero. I think this may be becase NVMe SSD has a completely new storage stack rather than conventional devices (e.g., SATA SSD).

Marco Bonelli
  • 63,369
  • 21
  • 118
  • 128
Tim He
  • 350
  • 2
  • 13
  • I find that maybe I can monitor specific data structure in the kernel: https://stackoverflow.com/questions/72508189 but the problem is -- must I modify the kernel and recompile it? – Tim He Jun 05 '22 at 14:36

1 Answers1

3

Solution:

cd /sys/kernel/debug/tracing/events/nvme/nvme_sq
# filter by disk name:
echo 'disk=="nvme0n1"' > filter
# enable the event:
echo 1 > enable
# check results from trace_pipe:
cat /sys/kernel/debug/tracing/trace_pipe

I suggest also enable /sys/kernel/debug/tracing/events/nvme/nvme_setup_cmd, then, you can briefly understand what is the nvme driver doing.

          <idle>-0       [002] d.h.  2558.073405: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=76, tail=76
   systemd-udevd-3805    [002] ....  2558.073454: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=3, cmdid=48, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=104856608, len=7, ctrl=0x8000, dsmgmt=7, reftag=0)
          <idle>-0       [002] d.h.  2558.073664: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=77, tail=77
   systemd-udevd-3805    [002] ....  2558.073704: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=3, cmdid=49, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=104856648, len=7, ctrl=0x8000, dsmgmt=7, reftag=0)
          <idle>-0       [002] d.h.  2558.073899: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=78, tail=78
   systemd-udevd-3805    [002] ....  2558.073938: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=3, cmdid=50, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=104854512, len=7, ctrl=0x8000, dsmgmt=7, reftag=0)
          <idle>-0       [002] d.h.  2558.074134: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=79, tail=79

The explanation of each field in this output can be found here.

Tim He
  • 350
  • 2
  • 13