ASP NET not using all available cores - Parallel.ForEach

Question

Our ASP NET software has this specific condition where Parallel.ForEach is applied. It was working fine, but we recently noticed that it stopped using all available cores for these specific heavy operations where parallelism is applied.

This is the code we use:

Parallel.ForEach(var1, new ParallelOptions
    { MaxDegreeOfParallelism = Environment.ProcessorCount - 1 }, var2 => ...

What we used to see at Task Manager was a high usage (like 90%) of our 32x cores server while running these specific heavy operations. But now it keeps itself using only one core (~5% only). What used to take 20-40 min now is taking 5-6 hours!

While on our computer (running locally with Visual Studio) we do see that all cores are working as expected (~80-90% usage).

Details: Server has 2x CPUs, x64, Windows Server 2012 bare metal (no VM). There are no hardware problems. Software runs at ASP NET 4.6.2. MaxDegreeOfParallelism correctly returns 32. IIS Manager / Advanced Settings: CPU limit = 900. Maximum Worker Process seems to have no difference. Database doesn't seem to be the problem because all data are previously stored at memory (lists variables at the code).

As the code used to work fine at this same server, and currently works fine on our localhost, we suspect that something changed at the server side.

We suspect that a Microsoft update may be the one causing this, although we didn't found anything so far. These are the latest 2022 updates that we suspect:

2022-05 Security and Quality Rollup for .NET Framework 3.5, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5013871). Installation date: ‎25/‎05/‎2022 09:43
2022-04 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5012330). Installation date: ‎20/‎04/‎2022 15:53
2022-02 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5010582). Installation date: ‎11/‎02/‎2022 20:27
2022-01 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5009720). Installation date: ‎12/‎01/‎2022 21:48

Does anyone know something about this issue? Any ideas on how to troubleshoot this?

Where is this `Parallel.ForEach` code located? Is it part of rendering a web page? Or it's part of an independent Console application or Windows service? — Theodor Zoulias, Jun 03 '22 at 07:39
First I'd try to check the core affinity of the process. Maybe somehow the affinity is getting set to one core? Then I'd try to spawn 31 tasks or threads manually instead of code above to see if the problem is in parallel.foreach or not. — Petrusion, Jun 03 '22 at 11:52
@Petrusion How do you check the core affinity? As I understand, the IIS Manager itself goes to other cores as needed. — rd1218, Jun 03 '22 at 12:08
@rd1218 In the Windows Task Manager, you can go to the Details tab, where you can right click on a process, click Set Affinity and see the ticked cores. Or you can read it in C# at runtime with `Process.GetCurrentProcess().ProcessorAffinity` — Petrusion, Jun 03 '22 at 12:11
@Petrusion Ok, good one. At task manager the w3wp (IIS worker process) is assigend to all cores. — rd1218, Jun 03 '22 at 13:03

score 1 · Accepted Answer · answered Jun 03 '22 at 12:20

Any ideas on how to troubleshoot this?

You could try running this experiment on your server, as a standalone Console application:

ParallelOptions options = new()
{
    MaxDegreeOfParallelism = Environment.ProcessorCount
};
int concurrency = 0;
Parallel.ForEach(Enumerable.Range(1, 100), options, item =>
{
    var current = Interlocked.Increment(ref concurrency);
    try
    {
        Console.WriteLine($"Processing {item}, Concurrency: {current}");
        for (int i = 0; i < 1_000_000_000; i++) { }
    }
    finally { Interlocked.Decrement(ref concurrency); }
});

The expected behavior is to see the CPU maxing out (100%), and the reported concurrency reaching the value 32.

Thank you for this, it actually got 100% of the cores. This showed me that there are no problems with .NET or Windows Server/IIS. Therefore the problem must be on the code itself. — rd1218, Jun 03 '22 at 16:39

ASP NET not using all available cores - Parallel.ForEach

1 Answers1