How to benchmark a few lines of C programming code?

Question

I recently heard of the idea of branchless programming and I want to give it a try and see if it can boost performance. I have the following C function.

int square(int num) {
    int result = 0;
    if (num > 10) {
        result += num;
    }
    return result * result;
}

After removed the if branch, I have this:

int square(int num) {
    int result = 0;
    int tmp = num > 10;
    result = result * tmp + num * tmp + result * !tmp;
    return result * result;
}

Now I want to know whether the branchless version if faster. I searched around and found a tool called hyperfine (https://github.com/sharkdp/hyperfine). So I wrote the following main function and test the two versions of the square function with hyperfine.

int main() {
    printf("%d\n", square(38));
    return 0;
}

The problem is that based on the hyperfine result, I can't determine which version is better. In C programming, how does people usually determine which version of a function is faster?

Below is some of my hyperfine result.

C:\my_projects\untitled>hyperfine branchless.exe
Benchmark #1: branchless.exe
  Time (mean ± σ):       5.4 ms ±   0.2 ms    [User: 2.2 ms, System: 3.2 ms]
  Range (min … max):     4.9 ms …   6.1 ms    230 runs

C:\my_projects\untitled>hyperfine branch.exe
Benchmark #1: branch.exe
  Time (mean ± σ):       6.1 ms ±   0.7 ms    [User: 2.2 ms, System: 3.7 ms]
  Range (min … max):     5.0 ms …   9.7 ms    225 runs

C:\my_projects\untitled>hyperfine branch.exe
Benchmark #1: branch.exe
  Time (mean ± σ):       5.5 ms ±   0.3 ms    [User: 2.1 ms, System: 3.5 ms]
  Range (min … max):     4.9 ms …   7.0 ms    211 runs

C:\my_projects\untitled>hyperfine branch.exe
Benchmark #1: branch.exe
  Time (mean ± σ):       5.6 ms ±   0.4 ms    [User: 2.0 ms, System: 3.9 ms]
  Range (min … max):     4.8 ms …   7.0 ms    217 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.


C:\my_projects\untitled>hyperfine branch.exe
Benchmark #1: branch.exe
  Time (mean ± σ):       5.7 ms ±   0.3 ms    [User: 1.9 ms, System: 4.0 ms]
  Range (min … max):     5.0 ms …   6.6 ms    220 runs

C:\my_projects\untitled>hyperfine branchless.exe
Benchmark #1: branchless.exe
  Time (mean ± σ):       5.6 ms ±   0.3 ms    [User: 1.9 ms, System: 3.9 ms]
  Range (min … max):     4.8 ms …   6.9 ms    219 runs

C:\my_projects\untitled>hyperfine branchless.exe
Benchmark #1: branchless.exe
  Time (mean ± σ):       5.8 ms ±   0.3 ms    [User: 1.5 ms, System: 4.0 ms]
  Range (min … max):     5.2 ms …   7.3 ms    224 runs

C:\my_projects\untitled>

Answer 1

How to benchmark a few lines of C programming code?

Compile the code and inspect generated assembly by your compiler.

It's typical to use godbolt and inspect the generated assembly there. Godbolt link.

A semi-not-really-reliable way is to count the assembly instructions executed. I do not know about windows - I work on linux. With gdb I use code presented in this question and with:

// 1.c
#if MACRO
int square(int num) {
    int result = 0;
    if (num > 10) {
        result += num;
    }
    return result * result;
}
#else
int square(int num) {
    int result = 0;
    int tmp = num > 10;
    result = result * tmp + num * tmp + result * !tmp;
    return result * result;
}
#endif
// start-stop places for counting assembly instructions
// Adding attribute and a specific asm syntax that is a GNU extension
// So that the compiler will not optimize the functions out
__attribute__((__noinline__)) void begin() { __asm__("nop"); }
__attribute__((__noinline__)) void finish() { __asm__("nop"); }
// trying to use volatile so that compiler 
// wouldn't optimize the function completely out
volatile int arg = 38, res;
int main() {
    begin();
    res = square(arg);
    finish();
}

Then compile and benchmark in bash:

# a short function to count number of instructions executed between "begin" and "finish" functions
$ b() { printf "%s\n" 'set editing off' 'set prompt' 'set confirm off' 'set pagination off' 'b begin' 'r' 'set $count=0' 'while ($pc != finish)' 'stepi' 'set $count=$count+1' 'end' 'printf "The count of instruction between begin and finish is: %d\n", $count' 'q' | gdb "$1" |& grep 'The count'; }

# then compile and measure
$ gcc -D MACRO=0 1.c ; b a.out
The count of instruction between begin and finish is: 34
$ gcc -D MACRO=1 1.c ; b a.out
The count of instruction between begin and finish is: 22

Looks like on my platform with gcc10 compiler without any options without optimizations the second version is executing 12 instructions short. But comparing compiler output with optimizations makes no sense. After enabling optimizations there is a difference of one instruction:

$ gcc -O -D MACRO=0 1.c ; b a.out
The count of instruction between begin and finish is: 11
$ gcc -O -D MACRO=1 1.c ; b a.out 
The count of instruction between begin and finish is: 10

Notes:

• with your code square(38) can be just optimized to no-op.
• with your code and hyperfine branchless.exe you are comparing the execution of printf ie. the time it takes to flush the output and print it, not the execution of square().
• As presented in that answer you can use a hardware counter when available.

Answer 2

As said in remark the execution time of printf is greater that the time you want to measure, and independently of that the time you try to measure is too small.

To have a measure you have to put square in a file and its call in an other file in a loop, also not using literals, else the generated code can be directly the result and nothing more (never underestimate the power of the optimization the compilers are able to do when they know all, e.g. C++ constexpr).

So for instance :

file c1.c

int square(int num) {
    int result = 0;
    if (num > 10) {
        result += num;
    }
    return result * result;
}

file c2.c

int square(int num) {
    int result = 0;
    int tmp = num > 10;
    result = result * tmp + num * tmp + result * !tmp;
    return result * result;
}

file main.c

#include <stdio.h>

extern int square(int);

int main(int argc, char ** argv)
{
  int n, v, r = 0;
  
  if ((argc == 3) && 
      (sscanf(argv[1], "%d", &n) == 1) &&
      (sscanf(argv[2], "%d", &v) == 1))
    while (n--)
      r += square(v);
  return r;
}

Using the first solution (without optimization) :

/tmp % gcc c1.c main.c 
/tmp % time ./a.out 1000000000 38
2.315u 0.000s 0:02.41 95.8% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
2.316u 0.000s 0:02.41 95.8% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
2.316u 0.000s 0:02.41 95.8% 0+0k 0+0io 0pf+0w
/tmp %

Using the second solution (without optimization) :

/tmp % gcc c2.c main.c 
/tmp % time ./a.out 1000000000 38
3.087u 0.000s 0:03.21 95.9% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
3.107u 0.000s 0:03.23 95.9% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
3.098u 0.000s 0:03.22 95.9% 0+0k 0+0io 0pf+0w
/tmp  %

So without optimization the second proposal needs more time, this is still the case even the difference is almost null between them compiling with optimization :

/tmp % gcc -O2 c1.c main.c
/tmp % time ./a.out 1000000000 38
1.337u 0.000s 0:01.39 95.6% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
1.336u 0.001s 0:01.39 95.6% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
1.343u 0.000s 0:01.39 96.4% 0+0k 0+0io 0pf+0w
/tmp % 
/tmp % 
/tmp % gcc -O2 c2.c main.c
/tmp % time ./a.out 1000000000 38
1.341u 0.000s 0:01.39 96.4% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
1.343u 0.000s 0:01.40 95.7% 0+0k 0+0io 0pf+0w
/tmp % time ./a.out 1000000000 38
1.339u 0.000s 0:01.39 95.6% 0+0k 0+0io 0pf+0w
/tmp %

I did under Linux but you can do the same under Windows using your tool to measure

For information the generated code with optimization are :

first way :

square:
.LFB0:
    .cfi_startproc
    movl    %edi, %edx
    xorl    %eax, %eax
    imull   %edi, %edx
    cmpl    $11, %edi
    cmovge  %edx, %eax
    ret

second way :

square:
.LFB0:
    .cfi_startproc
    xorl    %eax, %eax
    cmpl    $10, %edi
    setg    %al
    imull   %edi, %eax
    imull   %eax, %eax
    ret
    .cfi_endproc