List declaration speed differences in debug mode

Question

I have created a small benchmarking console app for C# using the System.Diagnostics.StopWatch class to observe how long 3 methods take:

How I run the Benchmark:

public void Run()
{
    Stopwatch sw = new Stopwatch();

    sw.Start();
    InitDirectly();
    sw.Stop();
    Console.WriteLine($"InitDirectly: {sw.Elapsed.TotalMilliseconds}ms");

    sw.Reset();

    sw.Start();
    InitWithAdd();
    sw.Stop();
    Console.WriteLine($"InitWithAdd: {sw.Elapsed.TotalMilliseconds}ms");

    sw.Reset();

    sw.Start();
    InitWithForLoop();
    sw.Stop();
    Console.WriteLine($"InitWithForLoop: {sw.Elapsed.TotalMilliseconds}ms");
}

Method 1:

private void InitListDirectly()
{
    var list = new List<string>() {
        "string",
        "string",
        "string",
        //... up to a 100th entry
    };
}

Method 2:

private void InitListViaAdd()
{
    var list = new List<string>();
    list.Add("string");
    list.Add("string");
    list.Add("string");
    //... up to a 100th entry
}

Method 3:

private void InitListViaForLoop() {
    var list = new List<string>();
    for (int i = 0; i < 100; i++) {
        list.Add("string");
    }
}

Now starting the stopwatch before Method 1, then stopping. Same for Method 2.

Method 1 took around ~0.800ms
Method 2 took around ~1.000ms
Method 3 took around ~0.120ms

Now I'm amazed, that the for-loop (Method 3) is so much faster. I expected Method 1 to be the fastest, since the other two methods have to call "Add(string)" and the third has to create a for-loop.

Why is Method 3 so much faster? Is that due to some compilermagic? Does it realize that the statement inside the for-loop will be identical for all it's iterations?

EDIT: I ran in debug mode.

Method1 and Method2 are internally identical by the way, the list initialiser syntax is just sugar around doing lots of `list.Add` calls. — DavidG, Feb 01 '19 at 13:59
Then what explains the second one being (marginally) slower? — Csharpest, Feb 01 '19 at 14:00
You should include the test because i assume that you are measuring incorrectly — Tim Schmelter, Feb 01 '19 at 14:00
I'm suspicious of your results though. Are you sure you benchmarked correctly? — DavidG, Feb 01 '19 at 14:00
Added the way i run the benchmark, also thanks for this "possible duplicate"-reference @Johnny , will look into it — Csharpest, Feb 01 '19 at 14:04
@Johnny this is not a duplicate of that post. OP is wondering why two former methods are way slower than later method (not asking for fastest way) — M.kazem Akhgary, Feb 01 '19 at 14:04
@M.kazemAkhgary they are discussing the same topic. For the benchmark I suggest to run more than 100 times and to run more than once and then take the average, if you run that only once it is not relevant. Also try to use Debug\Release and compare... — Johnny, Feb 01 '19 at 14:06
Are you running this code in release or debug mode? I suspect if you use release mode, both methods will take the same time. — DavidG, Feb 01 '19 at 14:07
Ive run it around 20 times and wrote down the average. Deviations are below around 100ms — Csharpest, Feb 01 '19 at 14:07
I have benchmarked this with [`bechmarkDotNet`](https://benchmarkdotnet.org/articles/overview.html) and the results where: [922.054ns],[921.826ns],[946.982ns] so all roughly equal. You have to compile in release, execute it without debugger (from console) and use warmups and take the average. — Tim Schmelter, Feb 01 '19 at 14:08
I ran this in release mode and the variation between the methods is virtually nothing., — DavidG, Feb 01 '19 at 14:10
I ran your code a million times and the total variation between 1 and 3 was 5ms. — DavidG, Feb 01 '19 at 14:14
Alright, i dig that, and could reproduce. My "benchmark" seems to account only for the first call of each method. After they got called once, the time to call it again reduces drastically. So still im unsure why the first call of the for-loop is way faster. — Csharpest, Feb 01 '19 at 14:26
Are you sure the first one is faster? Where are you running this code? — DavidG, Feb 01 '19 at 14:27
Release mode, VS2017 console application, called from main method. Im quite certain yes — Csharpest, Feb 01 '19 at 14:28
The [JIT](https://learn.microsoft.com/en-us/dotnet/standard/managed-execution-process#compiling_msil_to_native_code) would likely account for differences in speed when reproducing the test. — Wai Ha Lee, Feb 01 '19 at 14:34
Trying yo get meaningful performance numbers in a managed environment like .NET is not trivial. How did you account for the particular behavior of the JIT and CLR; how did you prime your runtime environment to get reproducible and meaningful measurements? If not accounting for how the JIT/CLR works, such "poor man's benchmarks" could easily be a case of not measuring what you think you measured. An example of how easy it is to get useless measurements: https://stackoverflow.com/questions/16471759/is-bitarray-faster-in-c-sharp-for-getting-a-bit-value-than-a-simple-conjuction-w/19027941#19027941 — , Feb 01 '19 at 14:47
Running methods once is wrong way to benchmark anything, there are many factors what affect single call. Running benchmark like you have 20 times and averaging results will still average wrong results. Instead of calling method once call it reasonably big enough N times (million is probably good), measure total time, divide by N. Now you can compare things which have similar overhead. — Sinatr, Feb 01 '19 at 15:04
Alright, I am very thankful, I will benchmark my stuff better the next time! you guys taught me well and pointed me into the right direction. ty! — Csharpest, Feb 01 '19 at 15:14

score 3 · Answer 1 · answered Feb 01 '19 at 14:49

3

It's not an answer to your question, but maybe if you use BenchmarkDotNet for tests your question would be different.

answered Feb 01 '19 at 14:49

BigMuzzy

138
4

score 1 · Answer 2 · answered Feb 02 '19 at 10:02

It took a whole millisecond to add 100 strings to a List<>? That's like 2 to 4 million clock cycles!! That's bonkers, and vastly longer than even debug-mode could explain for actually running the code.

Since you're calling each function exactly once, maybe that include the time to JIT-compile the function?

A short function with a loop could easily JIT into machine code much faster than the larger functions, even in debug mode, because there's just much less byte-code for the JIT compiler to work through.

Aha! this is a nice point, I will look into the JIT Compilation idea. Yeah the first call took almost a ms, subsequent calls took like 0,03 ms i think, so much less — Csharpest, Feb 04 '19 at 08:00

List declaration speed differences in debug mode

2 Answers2