Concurrency Delay When Performance testing on EF

Question

I'm running a Performance Testing on EF because we were having some issue running concurrent calls on our Server.

Here is my Tests which i'm executing against northwind database, with 20,000 extra employees in the database to slow down retrieval

single execution generally return in about 451 ms, however when calling in parallel it seems I'm seeing some weird extra execution time

When I Run 10 Concurrent calls I'm getting 4 times the over head. It may seem trivial but when I execute things on my database you can see it a whole lot worse.

When running sp_whoisactive you can see that sql has already finished and it is wait to send back the (246ms)ASYNC_NETWORK_IO

Sql is not bottle necked.

class Program
{
    public static int MAX_IO_THREADS = 2000;
    public static int MIN_IO_THREADS = 1000;
    public static int CONCURRENT_CALLS = 10;

    static void Main(string[] args)
    {
        ConfigureThreadPool();
        WarmUpContexts();

        ProfileAction();
        Console.ReadKey();
    }

    protected static void ProfileAction()
    {
        var callRange = Enumerable.Range(0, CONCURRENT_CALLS);

        var stopwatch = new Stopwatch();
        stopwatch.Start();

        var results = new ConcurrentDictionary<int, string>();
        var parallelOptions = new ParallelOptions { MaxDegreeOfParallelism = 512 };

        Parallel.ForEach(callRange, parallelOptions, i =>
        {
            var callSw = new Stopwatch();
            callSw.Start();
            var result = RunTest();
            callSw.Stop();
            results.TryAdd(i, $"Call Index: {i} took {callSw.ElapsedMilliseconds} milliseconds to complete");

        });

        stopwatch.Stop();
        var milliseconds = stopwatch.ElapsedMilliseconds;

        foreach (var result in results.OrderBy(x => x.Key))
        {
            Console.WriteLine(result);
        }

        Console.WriteLine("Total Time " + milliseconds);
    }

    public static void ConfigureThreadPool()
    {
        ThreadPool.GetMinThreads(out var defaultMinWorkerCount, out var _);
        ThreadPool.GetMaxThreads(out var defaultMaxWorkerCount, out var _);

        ThreadPool.SetMaxThreads(defaultMaxWorkerCount, MAX_IO_THREADS);
        var successMin = ThreadPool.SetMinThreads(defaultMinWorkerCount, MIN_IO_THREADS);
    }

    static void WarmUpContexts()
    {
        using (var stagingContext = new NORTHWNDEntities())
        {
            stagingContext.Employees.First();
        }
    }


    static int RunTest()
    {
        using (var context = new Entities())
        {
            var employees = context.Employees.ToArray();
            return employees.Length;
        }
    }
}

How can I optimize the performance of my threads so they actually return NOTE if I call async this will hang forever,

could this be a bug in .net?

Answer 1

A Few reasons why my project code was Delaying.

1.Configure Thread Pool. Opening up your IO threads will help delay with request that are not only cpu bound. Such as writing a file to disk, or a network bound request.

Note: if you're setting higher values you need to be careful not to have you min exceed the max, because these functions are poorly named, and should be called TrySetMin/TrySetMax since they return a bool whether successful or not

Be careful setting these I was only uses extremely high numbers for testing and learning purposes

ThreadPool.SetMaxThreads(defaultMaxWorkerCount, MAX_IO_THREADS);
ThreadPool.SetMinThreads(defaultMinWorkerCount, MIN_IO_THREADS);

2.GcServer Apparently, in .Net by Default the Garbage Collector is optimized for single Core performance instead of multicore Performance. You can read more on this issue on MSDN

Note: in .Net 4.0 Enabling GCServer Causing issues for application who had UI. In .Net 4.5 these issues were fixed by enabling gcConcurrent=true by Default

<configuration>  
   <runtime>  
      <gcServer enabled="true"/>  
   </runtime>  
</configuration>  

3.Additional Delays caused by EntityFrameWork, It looks like mapping relationships is really expensive, even if you are not pull out any related entities.

var employees = context.Employees.AsNoTracking().ToArray();

In was testing using an extremely large database, Moral is, if you are loading data for GET requests, if possible load without tracking.
Note: This may cause unexpected behavior as, different entities who share a common Foreign Key will have Different CLR object associated with them