Erratic average execution times for C function

Question

I'm trying to optimize a chunk of code given to me by a friend but my baseline for average execution times of it are extremely erratic and I'm lost to as why/how to fix it.

Code:

#include <sys/time.h>
#include <time.h>
#include <stdio.h>
#include "wall.h" /* Where his code is */

int main()
{
    int average;
    struct timeval tv;
    int i;

    for(i = 0; i < 1000; i++) /* Running his code 1,000 times */
    {
        gettimeofday(&tv, NULL); /* Starting time */ 

        start(); /* Launching his code */ 

        int ret = tv.tv_usec; /* Finishing time */ 
        ret /= 1000; /* Converting to milliseconds */ 
        average += ret; /* Adding to the average */ 
    }
    printf("Average execution time: %d milliseconds\n", average/1000);  
    return 0;
}

Output of 5 different runs:

804 milliseconds
702 milliseconds
394 milliseconds
642 milliseconds
705 milliseconds

I've tried multiple different ways of getting the average execution time, but each one either doesn't give me a precise enough answer or gives me a completely erratic one. I'm lost as to what to do now, any help would be greatly appreciated!

I know these types of benchmarks are very much system dependent, so I've listed my system specs below:

Ubuntu 12.10 x64
7.8 GiB RAM
Intel Core i7-3770 CPU @ 3.40GHz x 8
GeForce GT 620/PCIe/SSE2

Edit

Thank you all for your input but I decided to go with a gprof instead of constructing my own. Thank you, once again!

While it is true that benchmarks are specific to the hardware and operating system you run them on, it is even more dependent on the code you are evaluating. In this case, we can't see that code. Off hand, I would ask if there is any File IO happening. — Bill Lynch, Apr 17 '13 at 22:37
The `tv_usec` member of the `timeval` structure contains the microsecond part of the current time, not the number of microseconds since the Epoch. You are not measuring the execution time but rather sampling the microseconds part. The observed distribution of the value is quite normal (no pun intended). — Hristo Iliev, Apr 17 '13 at 22:37
Pardon my English, it is not "the microsecond part of the current time", but rather "the microsecond part of the time instant, when `gettimeofday()` was called". — Hristo Iliev, Apr 17 '13 at 22:49

score 5 · Answer 1 · edited May 23 '17 at 11:43

5

Your line int ret = tv.tv_usec; /* Finishing time */ doesn't give you the finishing time, it's still the starting time. You should make a second struct timeval, call gettimeofday with that and compare the two.

However, using clock() is probably easier. Of course, if you want to really analyse the performance of your code use a profiler.

edited May 23 '17 at 11:43

Community

1
1

answered Apr 17 '13 at 22:40

Kninnug

7,992
1
30
42

I just clicked "submit" ... to discover that you also posted, basically saying exactly the same thing. +1: I agree completely, on all points. IMHO... – paulsm4 Apr 17 '13 at 22:43
Yours is nice on finding the discouragement of `gettimeofday()` article though. ;) – Kninnug Apr 17 '13 at 22:47
Using `clock()` could be a bit tricky as the result depends on the OS. On most Unix systems it returns the used CPU time and not the elapsed wall-clock time, but on Windows it returns the wall-clock time. One should therefore use `clock_gettime(CLOCK_REALTIME, &tp);` where `tp` is of `struct timespec`. – Hristo Iliev Apr 17 '13 at 22:47
Doesn't dividing the result of `clock()` by the `CLOCKS_PER_SEC` macro give the elapsed time in seconds, regardless of OS? – Kninnug Apr 17 '13 at 22:51
No, it doesn't. It returns the CPU time used by all threads in the process on Linux and on BSD systems, including OS X. It is even written on the page that you have linked to. – Hristo Iliev Apr 17 '13 at 22:51
Tried and worked! Thanks! I decided to go with gprof in the end, though. But still, thanks! :) – centip3de Apr 17 '13 at 23:32

score 1 · Accepted Answer · answered Apr 17 '13 at 22:41

There are several problems here, including zero details on what the code you're benchmarking is doing, and that you're using "gettimeofday()" incorrectly (and, perhaps, inappropriately).

SUGGESTIONS:

1) Don't use "gettimeofday()":

http://blog.habets.pp.se/2010/09/gettimeofday-should-never-be-used-to-measure-time

2) Supplement your "time elapsed" with gprof:

http://www.cs.duke.edu/~ola/courses/programming/gprof.html

Erratic average execution times for C function

2 Answers2