10

I have big project written in C++. It might have some stability problems (i.e. random runtime), but I'm not sure about it. I understand that execution time, measured by wall clock time might be different among runs, because of OS multitasking. But I don't know, whether it's normal for stable program, to have varying execution time measured by cpu clock time among runs with same input. I tried to use clock() from time.h, and 

boost::chrono:::process_user_cpu_clock::now();

But in both cases I see spikes on a graph. I'll give you an example of such graphs. Here Y axis - execution time, X axis - consecutive runs of a same program, on same input data. Red graph - wall clock time, red - cpu clock time, taken by clock() from time.h

enter image description here

Of course we assume that our program is stable, and doesn't have any random behaviour. So, is it possible? Platform is Windows 7.

Ibraim Ganiev
  • 8,934
  • 3
  • 33
  • 52
  • 3
    What's the platform? Linux `clock()` is processor time, Windows `clock()` AFAIK is wall clock time. – Weather Vane May 25 '16 at 18:09
  • 4
    A graph is useless without units, labelled axes, scale, etc. Are you really worried about minute differences (in the range of 1.21 - 1.33, say milliseconds) for example? – uh oh somebody needs a pupper May 25 '16 at 18:13
  • 2
    @sleeptightpupper The y axis is in seconds. It says that in the key. I just assume x is each run. – NathanOliver May 25 '16 at 18:14
  • 3
    A tenth of a second is peanuts. The OS heading off to do something that's more important to it than running your code can inject all sorts of fun artifacts that slow you down by a lot more than 100ms. Recommend running your program through some real profiling software. – user4581301 May 25 '16 at 18:15
  • 1
    Check out http://stackoverflow.com/questions/17432502/how-can-i-measure-cpu-time-and-wall-clock-time-on-both-linux-windows – Andrew Komiagin May 25 '16 at 18:16
  • @WeatherVane, Windows 7. – Ibraim Ganiev May 25 '16 at 18:41
  • @sleeptightpupper, units here are seconds, but this graph indeed is not good, I just can't find good example, which will take more time to calculate. – Ibraim Ganiev May 25 '16 at 18:41
  • @NathanOliver, yes, X is each run. – Ibraim Ganiev May 25 '16 at 18:41
  • @user4581301, I know about multitasking :), I just asked whether it's normal for cpu clock time to be so random? Also, if you know some software which could help at finding such problems (Random execution time) - give me advices. – Ibraim Ganiev May 25 '16 at 18:42

3 Answers3

9

Of course we assume that our program is stable, and doesn't have any random behaviour. So, is it possible?

If you're program is running on a desktop, this variability is typical, and I would say unavoidable. Interrupts, i/o channel activity, and Ethernet itself consume cpu time, often with surprisingly large 'blocks-of-time' (see tcp / ip SAR, cache misses, etc), most of which is beyond your program's control and not in-synch with your timing efforts.

I have seen only one example of software running in a 'stable' way that you hint at. That computer was a SBC (single board computer), with 1 cpu (not Intel or AMD), all static ram (so no dynamic ram, and no refresh activity), no Ethernet, but two channels of i/o at fixed rate, and it ran a single program on a scaled down op system (not linux, not a desktop os) ... the precision was as if the behaviour was simple hw logic.

As team lead, I recognized the unusual, so I asked her if she had time to attach a logic analyzer and scope ... she demonstrated that neither tool showed any variance in time, edge to edge, message to message. Her software logic was, to me, impressively straight forward. In that system, if you did not need an interrupt, you simply did not enable it.

A desktop is a remarkably different beast ... so many things going on at the same time, most of which can not be stifled.

Yes. It is not only possible but unavoidable that a desktop has the kinds of variance (in timing) you are seeing.

And yet it is possible to achieve the stability you have hinted at, just not on a desktop. It takes special hardware, and careful coding.

2785528
  • 5,438
  • 2
  • 18
  • 20
3

OP says is using Windows 7, if that is MSVC the man page says

The clock function tells how much wall-clock time the calling process has used. Note that this is not strictly conformant with ISO C99, which specifies net CPU time as the return value. To obtain CPU time, use the Win32 GetProcessTimes function.

That is why the apparent execution time is inconsistent.

Weather Vane
  • 33,872
  • 7
  • 36
  • 56
3

This is absolutely normal. There are many effects that cause this. In fact repeatability for performance experiments is very difficult to achieve.

Cache / Memory

Any performance involving memory heavily depend on whether the cache is used or not:

The operating system schedules another thread on your core? Even if the time of this thread doesn't count for your process clock, it can evict your working data from the cache and your program runs slower afterwards.

The operating system decides to move your thread to another core? The core-local cache doesn't contain your working data. In a parallel application, the mapping from threads to cores has a huge impact on performance.

The operating system decides to run another memory-intense thread on in parallel (on a different core)? Your application has less shared cache and memory bandwidth available.

Hardware

On a recent system you likely use turbo mode - that changes CPU frequency based on many parameters including temperature.

Modern CPUs have many heuristics that effect performance. For instance a branch predictor guesses which branch you take for a jump based on past "experience". Your performance varies greatly whether that guess is right or not, as explained in detail in this epic answer. There are other components like that, for example prefetchers.

Clock accuracy

Clocks are not perfect either. They have limited resolution and accuracy. They can drift over time - or differ among cores. As indicated by Weather Vane, the software on top of the clocks can also be wrong.

This is by far a comprehensive list, just some examples.

Community
  • 1
  • 1
Zulan
  • 21,896
  • 6
  • 49
  • 109