How to test what method implementation runs faster

Question

While the question check if input is type of string has been closed two of the answers spiked a micro-optimization question in my mind: which of the below two solutions would perform better?

Reed Copsey provided a solution using Char.IsLetter:

string myString = "RandomStringOfLetters";
bool allLetters = myString.All( c => Char.IsLetter(c) );

Adapted solution using regex from Mark Byers:

string s = "RandomStringOfLetters";
bool allLetters = Regex.IsMatch(s, "^[a-z]+$", RegexOptions.IgnoreCase);

Not wanting to just ask the question of either Reed or Mark I thought I'd write a quick test to determine which performed better. Problem is I haven't done a lot of code optimization (I tend to put code readability above all else).

Other than taking a timestamp before and after the run of each, what are some other (better?) options of determining which solution runs faster?

Edit

I modified Martin's answer to work with Console.WriteLine(...) and ran it as a console application. Not sure exactly how LinqPad runs applications but the results were about the same:

41
178

Answer 1

You'll want to do this, measuring the runtimes using a Stopwatch. Also, here are a few very important things to keep in mind when profiling:

1. Always run your test more than 1 time. The first time you run it, there will be overhead from the JIT, and the timings may be misleading. Running many times and taking the average is a good approach (I'll often run a test like this 100,000 times, for example.)
2. Always run your test with a full Release build, outside of the Visual Studio hosting process. (By default, you can use Ctrl+F5 for this.) The Visual Studio host dramatically impacts timings.

Answer 2

you should check out System.Diagnostics.Stopwatch!

http://msdn.microsoft.com/en-us/library/system.diagnostics.stopwatch.aspx

You should run the thing many times in a loop to reduce timing errors and other uncontrollable factors.

Hope that helps.

Answer 3

I just put this together in LINQPad as an example of how I'd do it (hence the calls to Dump() - Replace with Console.WriteLine(...) if you aren't using this handy tool).

Looks like the LINQ way is slightly more than four times faster:

System.Diagnostics.Stopwatch stopwatch = new Stopwatch();

stopwatch.Start();
for (int i = 0; i < 100000; i++)
{
 string myString = "RandomStringOfLetters";
 bool allLetters = myString.All( c => Char.IsLetter(c) );
}
stopwatch.Stop();
stopwatch.ElapsedMilliseconds.Dump();

stopwatch.Reset();

stopwatch.Start();
for (int i = 0; i < 100000; i++)
{
 string s = "RandomStringOfLetters";
 bool allLetters = Regex.IsMatch(s, "^[a-z]+$", RegexOptions.IgnoreCase);
}
stopwatch.Stop();
stopwatch.ElapsedMilliseconds.Dump();

Output:

47 
196

Answer 4

There is a System.Diagnostics.Stopwatch class that can be used.

Whatever code you test, run the test once to remove JIT costs and then run it again for the final timings. Most individual timing counts could be unrepresentative due to other factors on the PC - so run many iterations and then calculate the average runtime from that.

Answer 5

Use the System.Diagnostics.Stopwatch class.

Start the StopWatch and run several thousand iterations, stop it and check the total milliseconds that have elapsed

Answer 6

Steps to determine which is faster:-

1. Get a collection of computers, a couple of hundred should do, AMD/Intel/other,32-bit/64-bit, ...

2. Install every .NET framework you care about on each of them (in turn)

3. Try each combination of optimization options for compilation (in turn)

4. Use StopWatch to test a large run for each

5. Monitor memory utilization for each as that may have a larger impact on the rest of your application. Saving a few cycles at the cost of increased memory consumption and more garbage collection activity is often a poor 'optimization'.

That might give you some idea about which is faster in practice, at least for current releases of the compiler. Repeat with each new release of the compiler.