2

I'm writing a high performance application (a raytracer) in C++ using Visual Studio, and I just spent two days trying to root out a performance drop I witnessed after refactoring the code. The reason it took so long was because the performance drop was smaller than the normal variation in execution time I witnessed from run to run.

Not sure if this is normal, but sometimes the program may run at around 33fps pretty consistently, then if you close and rerun, it may run at 37fps. This means that in order to test any new change, I had to manually run and rerun until I witnessed peak performance (And this could require up to like 10 runs). Simply running it for some large number of frames, and measuring the time doesn't fix this variability. For example, if the program runs for 40 seconds on average, it will nevertheless vary by over 1-2 seconds, which makes this test nearly useless for detecting the 1 millisecond per frame performance loss I was dealing with.

Visual Studio's profiling tools also didn't help find this small of an issue, because they also were subject to variation, and in any case, its not necessarily going to tell me the exact offending line, so I have to test solutions, and the profiler is not very effective at confirming a proposed solution's efficacy.

I realize this all may sound like premature optimization, but I don't think it is because I'm optimizing only after finishing complete features; I'm just trying to monitor changes in performance regularly so that issues like the above don't slip in and just get added to the apparent cost of the new feature.

Anyways, my question is simply whether there's a way to objectively determine the "real" speed of an application, discounting the effect of variation. Or, failing that, how do developers deal with such issues? I doubt that my current process is the ideal one.

user2345397
  • 301
  • 1
  • 11
  • I read somewhere else something like "_Test it 10 times with covering all the cases and then take the average time as your result_" – Khalil Khalaf Aug 10 '16 at 15:35
  • @FirstStep Thats a good idea; automating the process would at least speed it up considerably. And now that I think of it, the average of separate runs should be much more telling. My only worry is the question of where the variation comes from. If its due to the system periodically requiring more computational resources, then an average wouldn't necessarily converge to the correct value. I try to work without any unnecessary programs running, but there are limits to this, so obtaining a fair measurement still seems difficult. – user2345397 Aug 10 '16 at 15:42
  • How long does an individual test run take? I think I'd want 10 or 20 seconds of runs. (Possibly also worth timing a smallish constant program before and after to see if something is slowing the machine down.) – Martin Bonner supports Monica Aug 10 '16 at 15:47
  • If I automated my current method of testing, the individual tests would be pretty slow because I have to aquire an OpenGL context and load in data each time. But if I designed it so I could test different implementations during a single run, I'd avoid that and I'd avoid the strange variation in execution speed that happens between runs. And I like your idea about having a program that measures relative system speed. Maybe I could launch several threads to do this measurement and then use that to normalize the results of the tests... – user2345397 Aug 10 '16 at 16:01

1 Answers1

1

There are lots of profilers for both c++ and openGL. For those who just need the links, here are they.

OpenGL debugger-profiler

C++ profilers but I recommend Google orbit because it has dark theme.

My eyes stopped at

Objectively measure performance

As you mentioned the speed varies from run to run because it's too complex system. It helps if the scope is small and it only tests some key algorithms. It worth to automatize and collect some reference data. As every scientist say one test is not a test, you should rely on regular tests with controlled environments.

And here comes some tricks that can be used to measure performance.

  • In the comments others said, an average based on several runs may help you. It softens the noise from the outside.
  • Process priority or processor affinity could help you control the environment. By giving low priority to other processes your program gains more resource.
  • Measuring the whole execution of a test and compare it against processor time. As several processes runs at the same time, processor time may differs from execution time.
  • Update your reference values if you do a software update. Perhaps one update comes with performance boost while other with security patch.
  • Give a performance range for your program instead of one specific number. Perhaps the temperature messed up your measurement and the clock speed was decreased.
  • If a test runs too fast to measure, execute the most critical part several times in a test case. Too fast depend on how accurate you can measure. On ms basis it's really hard to decide if a test executed in 2 ms instead of 1 ms is a failure or not. However, if executed 1000 times - 1033 ms compared to 1000 ms gives you better insight.
  • Only test what is the critical section. Set up the environment and start the stopwatch when everything is ready. The system startup could be another test.
zerocukor287
  • 555
  • 2
  • 8
  • 23