Parallel.ForEach loop ThreadAbortException The Maximum Wait/Sleep/Join time has been exceeded

Question

I am using Parallel.ForEach loop to download multiple files from a Web API in parallel, in a .NET 4.8 application.
The problem is that sometimes I get a ThreadAbortException / The Maximum Wait/Sleep/Join time has been exceeded (00:05:00).
I can't seem to find a way to extend the "Maximum Wait Timeout" for the treads inside the Parallel.ForEach loop.

Sample code causing the problem.

// create the httpclient
using (HttpClient httpClient = new HttpClient())
{
    // now try to get documents in a parallel threaded task.
    Parallel.ForEach(documentIds, 
        new ParallelOptions { MaxDegreeOfParallelism = 
            Convert.ToInt32(Math.Ceiling((Environment.ProcessorCount * 0.50) * 2.0))},
        (documentId) =>
    {
        try
        {
            // post the data to the service
            using (HttpResponseMessage response =
                httpClient.PostAsync(serviceURL, new StringContent(
                requestBody.ToString())).GetAwaiter().GetResult())
            {
                // show error if service failed
                if (response.IsSuccessStatusCode == false)
                {
                    throw new Exception("Error getting content. " +
                        response?.ReasonPhrase);
                }

                // get back content
                using (Stream responseStream = response.Content.ReadAsStreamAsync()
                    .GetAwaiter().GetResult())
                {
                    // write the file to the FileSystem
                }
            }
        }
        catch (Exception ex)
        {
            // eat the error
            
            // NOTE: This does catch the ThreadAbortException but it breaks out
            // of the Parallel.ForEach BUT I want it to continue with other documents
        }
    });
}

Update 1
I got it to work but the solution is not a good one.
Basically the HttpClient is the problem.
If I create a NEW HttpClient inside the Parallel.ForEach loop, then it will let me handle the ThreadAbortException without breaking out of the loop.
MS best practices says to not create multiple HttpClients else it can keep to many open sockets before cleanup.

Answer 1

That's a misuse of Parallel.ForEach. That method is simply not meant for concurrent operations, it's only meant for in-memory data parallelism. This code blocks all CPU cores spinwaiting while waiting for responses. That means it pegs the CPU at 100% for a while before the threads start getting evicted.

Use Parallel.ForEachAsync instead, or Dataflow ActionBlock, eg :

await Parallel.ForEachAsync(documentIds, async (documentId,ct)=>{
...
});

This will use roughly as many worker tasks as there cores to process the data. You can specify a different number (higher or lower) with the ParallelOptions parameter

await Parallel.ForEachAsync(documentIds, 
    new ParallelOptions { MaxDegreeOfParallelism = 4},
    async (documentId,ct)=>{
    ...
    });

You may want to specify a lower number if the files are large, flooding the network, or you can increase it if there are a lot of smaller files, or the response takes a lot time to start.

ActionBlock - .NET Framework 4.8

Another option, one that can scale to an entire processing pipeline, is to use a Dataflow ActionBlock to process data concurrently.

var dop=new ExecutionDataflowBlockOptions
         {
            MaxDegreeOfParallelism = 10,
            BoundedCapacity=10
         };
var httpClient = new HttpClient()
var downloader=new ActionBlock<string>(docId=>DownloadAsync(httpClient,docId),dop);

Once you have the downloader block you can start posting doc IDs to it. When done, you tell the block to complete and await for all pending messages to finish processing:

foreach(var docId in documentIds)
{
    await downloader.SendAsync(docId);
}
downloader.Complete();
await downloader.Completion;

The downloading method itself can be fully asynchronous:

async Task DownloadAsync(HttpClient client,string documentId)
{
    var url=...;
    var req=new StringContent(requestBody.ToString());
    var response=await client.PostAsync(serviceURL, req);
    ...
    using var dlStream=await response.Content.ReadAsStreamAsync();
    using var fileStream=File.Create(somePath);
    await dlStream.CopyTo(fileStream);
}

All Dataflow blocks have an input buffer to hold input messages. By default, it has no limit. Adding BoundedCapacity=10 will cause the posting code to pause if more than 10 messages are pending. This can be used to throttle publishing code that's faster than the processing code and avoid filling up memory