C++ (Intel) vs Java (Hotspot) vs C# benchmark questions (code and results included)

Question

I have been comparing the raw CPU performance speed between the three main languages (code and results are below). I am very curious as to how the main languages compare for raw computational power. I had a theory that Java and C# could possibly rival C++ when there was no memory overhead involved.

My questions:

1) Edited (C++ timings now more realistic)

2) Am I right in thinking the JVM took ages on the first iteration, but for the second it had finished analysing and therefore optimised? How did Hotspot know to finish optimising after the first iteration of my outside loop and not halfway through?

3) Why does C# not perform like Java and heavily optimise at the start? What is different about C# with regards to Java? Why is the C# slower- is it simply due to less optimization?

4) Is there any specific reason why the oscillation between 2246 and 2262 milliseconds for the C# test timings, could this be two different times because the CPU has two cores?

EDIT: Updating code to show stopwatch usage in C# code.

EDIT: Correct C++ timing code and results

The setup:

• C++: VS2010 and Intel Compiler (built in release mode, Optimization: O2, Enable intrinsic functions: yes, favor size nor speed: neither, omit frame pointers: No, enable fiber-safe optimizations: no, whole program optimization: yes)

• Java: Eclipse, Hotspot 64 bit compiler version 17, Java 1.6

• C#: VS2010 and .net 4.0 (built in release mode)

• CPU: Intel E6600 (2.4GHz) running at 2.7GHz, bus speed 300MHz, 8GB memory, DRAM Freq: 375MHz

• Win 7 (64 bit)

C++ code:

#include "stdafx.h"
#include <iostream>
#include <stdio.h>
#include <windows.h>
#include <mmsystem.h>
#include <stdio.h>
#include <fstream> 

using namespace std;


double PCFreq = 0.0;
__int64 CounterStart = 0;

void StartCounter()
{
    LARGE_INTEGER li;
    if(!QueryPerformanceFrequency(&li))
        cout << "QueryPerformanceFrequency failed!\n";

    PCFreq = li.QuadPart;

    QueryPerformanceCounter(&li);
    CounterStart = li.QuadPart;
}
double GetCounter()
{
    LARGE_INTEGER li;
    QueryPerformanceCounter(&li);
    return double(li.QuadPart-CounterStart)/PCFreq;
}

static long counter = 0;

int _tmain(int argc, _TCHAR* argv[])
{

    for (int m = 0; m < 10; m++)
    {
        StartCounter();
        counter = 0;

        for (int j = 0; j < 3; j++)
        {
            //Just to test timing is working correctly
            //int* p = new int;

            for (long i = 0; i < 200000000; i++)
            {
                counter++;
            }
        }

        cout << GetCounter()*1000000 << " microseconds" << endl;
    }


    int p = 0;
    cin >> p;
    return 0;
}

C++ results:

7.19 microseconds

1.89

2.27

1.51

4.92

10.22

10.22

9.84

9.84

10.6

Java code:

public class main {

    static long counter = 0;

    public static void main(String[] args) {

        for(int m=0; m<10; m++){
            long start = System.nanoTime();
            counter = 0;

            for(int j=0;j<3; j++){
                for(long i=0; i<200000000; i++){
                    counter++;
                }
            }

            System.out.println(((System.nanoTime()-start)/1000000) + " ms");
        }
    }
}

Java results:

5703 milliseconds
471 ms
468 ms
467 ms
469 ms
467 ms
467 ms
467 ms
469 ms
464 ms

C# code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics

namespace t1
{
    class Program
    {
        static long counter = 0;

        static void Main(string[] args)
        {
            for (int m = 0; m < 10; m++)
            {
                Stopwatch s = new Stopwatch();
                s.Start();
                counter = 0;

                for (int j = 0; j < 3; j++)
                {

                    for (long i = 0; i < 200000000; i++)
                    {
                        counter++;
                    }

                }
                s.Stop();
                Console.WriteLine(s.Elapsed.TotalMilliseconds + " ms");
            }

            Console.ReadLine();
        }
    }
}

C# results:

2277 milliseconds

2246 ms

2262 ms

2246 ms

2262 ms

2246 ms

2262 ms

2246 ms

2262 ms

2262 ms

Answer 1

You've got a logical problem in your C++ code which uses QueryPerformanceFrequency:

PCFreq = double(li.QuadPart)/1000000000.0; // <- this is not correct
PCFreq = li.QuadPart;                      // <- this is correct

You should just assign li.QuadPart to PCFreq and do your conversion to milliseconds or nanoseconds in your printing code:

// convert from seconds to milliseconds
cout << GetCounter() * 1000.0 << endl;

With this change I get actual timings for your C++ code. Whether or not these timings are "valid" or useful in making comparisons, I will not comment.

Answer 2

1 - Sixlettervariables seems to have pointed out your mistake on this one

2 - The hotspot will optimize the code. Here is a similar question which also sees 10x speedups on loops. So what you see is expected output. First time a Java loop is run SLOW, why? [Sun HotSpot 1.5, sparc]

3 - I don't know enough about C# on this one to help. It's possible it doesn't optimize inner loops (you have 3 different loops). Maybe try extract the 2 loops you are testing into a completely separate method and see if that helps.

4 - DateTime is to represent a date and time, not high-precision timings. Hence it's not that accurate. DateTime.Now has a 10ms resolution as far as I'm aware

(FYI, this post has some good explanations on the JIT, C# and C++ optimizations which may help you: C++ performance vs. Java/C#)

Answer 3

I think it may be possible, that the compiler will calculate the value of counter at compiletime and does not iterate through your loop.

I think a simple counter is a really bad benchmark.

By the way, try to run the java-code within a method. It may be faster because of the JIT-optimization. (But I am not sure)

Answer 4

Benchmarks for the various compilers has already been done and put together very nicely.

The Computer Language Benchmark Games

Java 7 Server vs. GNU C++

C# Mono vs. GNU C++

C# Mono vs Java 7 Server

While Java 7 Server is faster than C# Mono 2.10.8 look at the amount of memory that Java 7 utilizes.