0

I have been searching for an appropriate method to measure cost of various syscalls in the Linux OS. There have been many questions raised related to this topic in the past, none provide a detailed description of how to measure it accurately. Most of the answers arbitrarily claim the cost of the syscall is 1-2us or a few 100 cycles if it caches on the CPU.

  1. System calls overhead
  2. Syscall overhead

The naive way I can think of measuring the syscall cost is to use rdtscp instruction across a syscall such as getpid(). However this is insufficient for measuring the cost of open(), read() or write() calls accurately. I do can modify the kernel and insert specific timer code across these functions and measure it but that would require changes in the kernel which I don't want to do. I wonder if there is a simpler solution that would allow me to measure it from the userspace itself.

Update: July 14: After a lot of searches, I found libmicro benchmark suite from RedHat. https://github.com/redhat-performance/libMicro

However, this is created a while ago and I am wondering how good this still is. Of course, it does not use rdtscp and that adds some measurement errors. Is there anything else that is missing in this benchmark creation?

vsm
  • 9
  • 1
  • 4
  • 1
    You mention two different things: 1. the cost of implementing an operation as a syscall compared to a function call (the syscall overhead), and 2. the cost of the whole syscall (the throughput of e.g. read/write calls). Which is it you're interested in? – that other guy Jul 13 '20 at 22:42
  • @that other guy, both actually. In terms of read/write syscall - I am more interested software overhead and not the time spent on reading the disk. – vsm Jul 15 '20 at 01:44
  • After a lot of searches - I found https://github.com/redhat-performance/libMicro Any thoughts on how good this is? – vsm Jul 15 '20 at 01:46
  • Note that Meltdown / Spectre mitigation adds *significant* overhead relative to the cost of fast system calls like `getpid` (if you actually call into the kernel for it instead of using the VDSO in user-space) or an invalid `syscall` that returns `-ENOSYS`. See [Fastest Linux system call](//stackoverflow.com/q/48913091) Answers from before 2018 need to be taken with a large grain of salt; much of the cost of a round trip into the kernel (without doing any work there) is microcode to clear branch-prediction. Also the resulting slower performance for some time after returning to userspace – Peter Cordes Jul 15 '20 at 02:42
  • "adds significant overhead relative to the cost of fast system calls" This is what I am interested to measure. I know people keep stating "significant" overhead but I could not measure it myself. The idea I have is to measure with and without mitigation to see the difference to find the cost of meltdown mitigation. – vsm Jul 15 '20 at 04:28

1 Answers1

2

strace and perf are generally used to track and measure such kind of (kernel) operations. More specifically, perf can be used to generate flame graphs enabling you to see detailed in-kernel function calls. However, one should remember that proper rights need to be adjusted in /proc/sys/kernel/perf_event_paranoid.

I advise you to put the syscall in a loop since measuring precisely the cost of one syscall with possibly delayed/asynchronous work affected to kernel threads is either very hard to measure in user-space or simply just inaccurate (on a non-customized kernel).

Additional information:

strace work at the microsecond granularity. Some the POSIX clocks (see clock_gettime) could reach the granularity of 100 ns. Beyond this limit, rdtscp is AFAIK one of the most accurate (one should care about the reference frequency). As for perf, it makes use of hardware performance counters and kernel events. You may need to configure your kernel so trace-points can be generated and properly tracked by perf. perf can track one specific process or the complete system.

Jérôme Richard
  • 41,678
  • 6
  • 29
  • 59
  • Thanks! I added an additional update to the question. Any thoughts on that? – vsm Jul 15 '20 at 01:49
  • 2
    *although care must be taken to avoid frequency scaling* - On any still-relevant CPUs, the TSC ticks at a fixed reference frequency. That's *not* necessarily the same as the max non-turbo frequency (e.g. 4008 MHz on a 4GHz i7-6700k, or more significantly different on some Ice Lake CPUs). ([see this](https://stackoverflow.com/a/51907627)). So avoiding idle/turbo isn't sufficient to use TSC = core clocks. You probably do want to keep core clock frequency constant somehow, though, for repeatable measurements. i.e. do a warm-up delay loop to get the CPU up to speed before the timed region. – Peter Cordes Jul 15 '20 at 02:34
  • @PeterCordes Indeed. Very good point! Thank you for this information. – Jérôme Richard Jul 15 '20 at 08:27
  • @PeterCordes Agreed. Thanks for the info. – vsm Jul 30 '20 at 22:00