How can I gather results from parallel async tasks in C#?

Question

I'm actually facing a problem with parallelism on asynchronous tasks.

My goal is to have an async Task which connects to multiple endpoints to retrieve the same kind of data. A list with all "connection strings" is provided. The data retrieval shall happen in parallel with all connections and the retrieved data of each connection shall be merged into a single container which will be returned, once all parallel async tasks finished.

My first idea was using AsParallel.ForAll() - passing the configurations, and an async Task to fill a (Dataflow) BufferBlock with SendAsync. Now I got stuck on the problem, that Parallel queries seem to only take Action<T> as parameter and not Task<T>. This will lead to an undesired behavior if an exception occurs.

Does anyone have a better approach to my problem?

Here is a snippet for better visualization:

configurations.AsParallel().ForAll(async config => await FetchAllData(config,
    resultBufferBlock, cancellationToken));

After some comments: I basically need Parallel.ForEachAsync which is available in .NET 6. we will move to .NET 6 in a couple of months. This is already planned. Although I would appreciate a solution for .NET 5. Actually it looks like I will use the bad way using the task as Action and fix that later…

Is this what you are looking for ? `Parallel.ForEachAsync` https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.parallel.foreachasync?view=net-6.0 — maraaaaaaaa, Feb 02 '22 at 18:32
Is this helpful? [How to limit the amount of concurrent async I/O operations?](https://stackoverflow.com/questions/10806951/how-to-limit-the-amount-of-concurrent-async-i-o-operations) Or this? [Parallel foreach with asynchronous lambda](https://stackoverflow.com/questions/15136542/parallel-foreach-with-asynchronous-lambda) — Theodor Zoulias, Feb 02 '22 at 18:32
@maksymiuk I recently learned the `Parallel.ForEachAsync` is .net 6 _(it is tagged .net 5)_ — Jeroen van Langen, Feb 02 '22 at 18:35
@maksymiuk yes! Didn't come accross this because it's .Net 6 and currently we are on .Net 5. I'm already happy this problem is solved in .Net 6. Anyway I need a solution before we move to .NET 6 (we will probably do it in a couple of months). — Otto V., Feb 02 '22 at 18:36
A custom implementation of the `Parallel.ForEachAsync` (one of many) can be found [here](https://stackoverflow.com/questions/11564506/nesting-await-in-parallel-foreach/65251949#65251949). — Theodor Zoulias, Feb 02 '22 at 19:18
There's also the actual source code for the .NET 6 implementation of ForEachAsync: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.ForEachAsync.cs — Gabriel Luci, Feb 02 '22 at 19:24

score 3 · Accepted Answer · answered Feb 03 '22 at 05:04

I'm actually facing a problem with parallelism on asynchronous tasks.

The problem you describe is concurrency (doing more than one thing at a time) and AFAICT does not need parallelism (multiple threads).

Does anyone have a better approach to my problem?

The way to do asynchronous concurrency is to start all the tasks (commonly using a LINQ select with an asynchronous lambda), and then use Task.WhenAll to join them.

var tasks = configurations.Select(async config => await FetchAllData(config, resultBufferBlock, cancellationToken));
await Task.WhenAll(tasks);

After some comments: I basically need Parallel.ForEachAsync which is available in .NET 6.

I disagree. Parallel.ForEachAsync is for when you have a complex mixture of CPU-bound and I/O-bound code and need to do both parallelism and asynchronous concurrency. In this case parallelism isn't necessary.

How can I gather results from parallel async tasks in C#?

1 Answers1

Linked