How do I retrieve the processor time in Linux without function calls?

Question

I need to calculate the running time of a portion of (C++) code and want to do this by finding the number of clock ticks elapsed during the execution of the code.

I want to find the processor time at the beginning of the code and the processor time at the end and then subtract them to find the number of elapsed ticks.

This can be done with the clock function. However, the time I'm measuring needs to be very precise and using a function call proved to be very intrusive since the caller-saved register allocator spilled many variables on each call.

Therefore, I cannot use any function calls and need to retrieve the processor time myself. Assembly code is fine.

I am using Debian and an i7 Intel processor. I can't use a profiler because it's too intrusive.

Answer 1

You should read time(7). Be aware that even written in assembler, your program will be rescheduled at arbitrary moments (perhaps a context switch every millisecond; look also into /proc/interrupts and see proc(5)). Then any hardware timer is meaningless. Even using the RDTSC x86-64 machine instruction to read the hardware timestamp counter is useless (since after any context switch it would be wrong, and the Linux kernel is doing preemptive scheduling, which does happen at any time).

You should consider clock_gettime(2). It is really fast (about 3.5 or 4 nanoseconds on my i5-4690S, when measuring thousands of calls to it) because of vdso(7). BTW it is a system call, so you might code directly the assembler instructions doing them. I don't think it is worth the trouble (and could be slower than the vdso call).

BTW, any kind of profiling or benchmarking is somehow intrusive.

At last, if your benchmarked function runs very quickly (much less than a microsecond), cache misses become significant and even dominant (remember that an L3 cache miss requiring effective access to DRAM modules lasts several hundred nanoseconds, enough to run hundreds of machine instructions in L1 I-cache). You might (and probably should) try to benchmark several (hundreds of) consecutive calls. But you won't be able to measure precisely and accurately.

Hence I believe that you cannot do much better than using clock_gettime and I don't understand why it is not good enough for your case... BTW, clock(3) is calling clock_gettime with CLOCK_PROCESS_CPUTIME_ID so IMHO it should be enough, and simpler.

^{In other words, I believe that avoiding any function calls is a misconception from your part. Remember that function call overhead is a lot cheaper than cache misses!}

See this answer to a related question (as unclear as yours); consider also using perf(1), gprof(1), oprofile(1), time(1). See this.

At last, you should consider asking more optimizations from your compiler. Have you considered compiling and linking with g++ -O3 -flto -march=native (with link-time optimizations).

If your code is of numerical and vectorial nature (so obviously and massively parallelisable), you could even consider spending months of your development time to port its core code (the numerical compute kernels) on your GPGPU in OpenCL or CUDA. But are you sure it is worth such an effort? You'll need to tune and redevelop your code when changing hardware!

You could also redesign your application to use multi-threading, JIT compilation and partial evaluation and metaprogramming techniques, multiprocessing or cloud-computing (with inter-process communication, such as socket(7)-s, maybe using 0mq or other messaging libraries). This could take years of development. There is No Silver Bullet.

^{(Don't forget to take development costs into account; prefer algorithmic improvements when possible.)}

Answer 2

Would a profiling tool allow me to measure the run time of a specific loop for instance?

Put your loop in an executable by itself and run perf stat ./a.out to time it and count cache-misses, branch-misses, and whatever else. Wrap a repeat loop around it if it's too short. Then you have a microbenchmark that you can profile with hardware performance counters to analyze how it decodes to uops, whether they schedule as efficiently as you thought, and so on.

Check the asm output to make sure it doesn't optimize anything away, or optimize across iterations of the outer repeat loop. In C you can use asm volatile("" ::: "memory"); as a compiler memory barrier, or you can use asm("" : "+g"(my_variable)) to force the compiler to have a C variable in a register at each step, with an empty asm statement as part of the dependency chain.

See my answer on Can x86's MOV really be "free"? Why can't I reproduce this at all? for an example using pure asm to make a static binary. (Negligible startup overhead so perf counters for the whole executable have less noise. But if you run it for 1000 ms and startup only takes a couple microseconds, you have very high precision and repeatability in your measurements).

It's still a microbenchmark though, measuring your loop in isolation with caches and branch prediction primed, not like what happens when your loop runs occasionally in the context of your whole program. Don't let this fool you into unrolling it too much; that wins on microbenchmarks but can hurt in practice.

Also, I think a profiler would use many function calls which would again create a lot of noise in my data.

Profilers that use HW performance counters are very non-intrusive.

Using perf record ./main-program / perf report -Mintel will show you a statistically sampled map of hot asm instructions in your code. (But keep in mind that cycles usually get charged to the instruction waiting for an input, rather than the instruction that was slow to produce it.)

Answer 3

On x86, you can do this using the rdpmc instruction, which allows you to read performance monitor counters from user space¹. You don't need any function calls to use this: you can insert the rdpmc instruction inline with the code you want to measure.

In particular, one of the counters you can use is CPU_CLK_UNHALTED.CORE which will allow you to measure small segments of code down to cycle accuracy.

If you can isolate the part of the program you want to measure into a small benchmark, you can use a Linux-compatible benchmark program I am writing, uarch-bench, to time your code and it will take care of most of the complexity of setting up the counters, running the benchmark repeatedly and excluding outliers.

The benchmark described above uses libpfc to read the Intel PMU from userspace. It was created specifically to address scenarios like this. If you want to measure the cycles of some portion of code directly inside your existing process, you can use libpfc by itself, using the PFSTART and PFCEND macros around the code you want to measure. These are macros that inline the measurement code directly: it doesn't involve any function call.

---

¹ At least as long as CR4.PCE is set, which is relatively straightforward on Linux.

Answer 4

You have been given a lot of reasons that what you want to do isn't going to give you good results, but no one has mentioned how to do what you are asking so you can see for yourself.

static inline unsigned long long rdtsc()
{
    uint32_t a, d;
    asm volatile("rdtsc" : "=a"(a), "=d"(d));
    return (unsigned long long)d << 32 | a;
}

rdtsc is not serializing, so to prevent it from reordering in hardware (and executing before previous instructions are complete), you may want to use rdtscp or lfence; rdtsc .

lfence; rdtsc on your i7 also stops later instructions from starting before it runs. See Difference between rdtscp, rdtsc : memory and cpuid / rdtsc?.