Why wouldn't I just run my performance benchmarks without debugging instead of "warming up" the code?

Question

I don't quite understand why you would just run your program without the debugger attached instead of running the test code without timing, then running the same code with timing. Sure this gives you a more even keeled test, but does it give you an absolute benchmark or is it just measuring raw performance?

It just seems that you aren't necessarily testing the performance as it would be run in a real world environment.

have a look here http://tech.pro/blog/1293/c-performance-benchmark-mistakes-part-one — Rune FS, Jun 18 '13 at 18:03
When the debugger is attached, the runtime has to check, for every line of code, whether it's targeted by any breakpoints. And probably does many more things that cause an overhead that would deform the results. When measuring things, you want to get rid of as many confounding factors as possible. — millimoose, Jun 18 '13 at 18:10

score 3 · Accepted Answer · answered Jun 18 '13 at 18:02

3

The first few times you execute some code, the C# runtime is going to optimize it in the background - this is known as "JIT", or "Just in Time compiling". So if the code you're measuring is meant to be repeated many times - say, in a loop that's running 24h/24, then you want to measure how long it takes after the JIT is done, since that's what's going to be running most of the time.

answered Jun 18 '13 at 18:02

redtuna

4,586
21
35

Doesn't the .NET runtime JIT everything when the assembly is loaded? – millimoose Jun 18 '13 at 18:03
2

@millimoose it's explained here http://tech.pro/blog/1293/c-performance-benchmark-mistakes-part-one – Rune FS Jun 18 '13 at 18:04
It's more progressive than that. See e.g. http://stackoverflow.com/questions/5589409/c-sharp-jit-compiling-and-net – redtuna Jun 18 '13 at 18:05
Also, this isn't what the OP actually asked about. – millimoose Jun 18 '13 at 18:08
1

Hey, at least I tried. Honestly the OP's question wasn't exactly super-clear. – redtuna Jun 18 '13 at 18:11
Actually @redtuna's link was helpful. It's obvious I don't know enough about the inner workings of the framework to ask a super intelligent question. – aledford Jun 18 '13 at 18:17
Yay! I'm glad it was helpful :) – redtuna Jun 18 '13 at 19:58

Why wouldn't I just run my performance benchmarks without debugging instead of "warming up" the code?

1 Answers1