How to await a list of tasks asynchronously using LINQ?

Question

I have a list of tasks that I created like this:

public async Task<IList<Foo>> GetFoosAndDoSomethingAsync()
{
    var foos = await GetFoosAsync();

    var tasks = foos.Select(async foo => await DoSomethingAsync(foo)).ToList();

    ...
}

By using .ToList(), the tasks should all start. Now I want to await their completion and return the results.

This works in the above ... block:

var list = new List<Foo>();
foreach (var task in tasks)
    list.Add(await task);
return list;

It does what I want, but this seems rather clumsy. I'd much rather write something simpler like this:

return tasks.Select(async task => await task).ToList();

... but this doesn't compile. What am I missing? Or is it just not possible to express things this way?

Do you need to process `DoSomethingAsync(foo)` serially for each foo, or is this a candidate for [Parallel.ForEach](http://msdn.microsoft.com/EN-US/library/dd992001(v=VS.110,d=hv.2).aspx) ? — mdisibio, Feb 19 '14 at 00:02
@mdisibio - `Parallel.ForEach` is blocking. The pattern here comes from Jon Skeet's [Asynchronous C# video on Pluralsight](http://pluralsight.com/training/Courses/Description/skeet-async). It executes in parallel without blocking. — Matt Johnson-Pint, Feb 19 '14 at 00:06
@mdisibio - Nope. They run in parallel. [Try it](https://gist.github.com/mj1856/9084379). (Additionally, it looks like I don't need `.ToList()` if I'm just going to use `WhenAll`.) — Matt Johnson-Pint, Feb 19 '14 at 01:24
Point taken. Depending on the how `DoSomethingAsync` is written, the list might or might not be executed in parallel. I was able to write a test method that was and a version that was not, but in either case the behavior is dictated by the method itself, not the delegate creating the task. Sorry for the mix-up. However, if `DoSomethingAsyc` returns `Task`, then the `await` in the delegate is not absolutely necessary...I think that was the main point I was going to try to make. — mdisibio, Feb 19 '14 at 01:45

Stephen Cleary · Accepted Answer · 2014-08-20T20:05:15.610

151

LINQ doesn't work perfectly with async code, but you can do this:

var tasks = foos.Select(DoSomethingAsync).ToList();
await Task.WhenAll(tasks);

If your tasks all return the same type of value, then you can even do this:

var results = await Task.WhenAll(tasks);

which is quite nice. WhenAll returns an array, so I believe your method can return the results directly:

return await Task.WhenAll(tasks);

edited Aug 20 '14 at 20:05

answered Feb 18 '14 at 23:57

Stephen Cleary

437,863
77
675
810

12

Just wanted to point out that this can also work with `var tasks = foos.Select(foo => DoSomethingAsync(foo)).ToList();` – mdisibio Feb 19 '14 at 01:50
1

or even `var tasks = foos.Select(DoSomethingAsync).ToList();` – Todd Menier Aug 20 '14 at 19:33
5

what is the reason behind it that Linq does'nt work perfectly with async code? – Ehsan Sajjad Sep 27 '16 at 21:05
3

@EhsanSajjad: Because LINQ to Objects synchrously works on in-memory objects. Some limited things work, like `Select`. But most don't, like `Where`. – Stephen Cleary Sep 28 '16 at 01:31
1

If we have in memory long running operation, is it good approach to use async await there? – Ehsan Sajjad Sep 28 '16 at 07:29
1

@EhsanSajjad: The use case for `async` is freeing up threads. So, if you want to free up the UI or ASP.NET request thread, then yes. – Stephen Cleary Sep 28 '16 at 09:24
1

and if that processing we are already doing in a background thread, then it's no use to put async await? – Ehsan Sajjad Sep 28 '16 at 09:39
5

@EhsanSajjad: If the operation is I/O-based, then you can use `async` to reduce threads; if it's CPU-bound and already on a background thread, then `async` wouldn't provide any benefit. – Stephen Cleary Sep 28 '16 at 10:27
So, what if I need each task to be await in sequence, as in the case with LINQ to EF? – Dave Van den Eynde May 08 '17 at 15:41
1

You just do one `await` at a time. EF does support `await`, but it doesn't support multiple simultaneous requests on the same context, so you can't do something like my answer here with `Select` followed by `WhenAll`. (Unless you use multiple db contexts). – Stephen Cleary May 08 '17 at 16:36

score 8 · Answer 2 · edited Dec 15 '15 at 11:31

8

To expand on Stephen's answer, I've created the following extension method to keep the fluent style of LINQ. You can then do

await someTasks.WhenAll()

namespace System.Linq
{
    public static class IEnumerableExtensions
    {
        public static Task<T[]> WhenAll<T>(this IEnumerable<Task<T>> source)
        {
            return Task.WhenAll(source);
        }
    }
}

edited Dec 15 '15 at 11:31

Peter Mortensen

30,738
21
105
131

answered Apr 08 '15 at 20:05

Clement

3,990
4
43
44

11

Personally, I would name your extension method `ToArrayAsync` – torvin Nov 15 '15 at 21:59

score 4 · Answer 3 · edited Oct 08 '21 at 11:28

One issue with Task.WhenAll is that it would create a parallelism. In most of the cases it might be even better, but sometimes you want to avoid it. For example, reading data in batches from DB and sending data to some remote web service. You don't want to load all the batches to the memory but hit the DB once the previous batch has been processed. So, you have to break the asynchronicity. Here is an example:

var events = Enumerable.Range(0, totalCount/ batchSize)
   .Select(x => x*batchSize)
   .Select(x => dbRepository.GetEventsBatch(x, batchSize).GetAwaiter().GetResult())
   .SelectMany(x => x);
foreach (var carEvent in events)
{
}

Note .GetAwaiter().GetResult() converting it to synchronous. DB would be hit lazily only once batchSize of events have been processed.

score 2 · Answer 4 · answered Feb 18 '14 at 23:54

2

Use Task.WaitAll or Task.WhenAll whichever is approriate.

answered Feb 18 '14 at 23:54

L.B

114,136
19
178
224

1

That doesn't work either. `Task.WaitAll` is blocking, not awaitable, and won't work with a `Task`. – Matt Johnson-Pint Feb 18 '14 at 23:55
1

@MattJohnson `WhenAll` ? – L.B Feb 18 '14 at 23:56
Yep. That's it! I feel dumb. Thanks! – Matt Johnson-Pint Feb 18 '14 at 23:57

score 1 · Answer 5 · answered Feb 18 '14 at 23:57

1

Task.WhenAll should do the trick here.

answered Feb 18 '14 at 23:57

Ameen

2,576
1
14
17

score 0 · Answer 6 · answered Oct 08 '21 at 06:30

0

Expanding on Stephen's answer it can also be expressed without .ToList() as follows:

var tasks = foos.Select(aFoo => aFoo.DoSomething());
await Task.WhenAll(tasks).ConfigureAwait(true);

Background: In some scenarios calling .ToList() can result in side effects executed at that time because the enumerable is then enumerated. If the enumerable is a call to a set of APIs or a set of queries, this may not be the desired behavior at that time. Without .ToList() the enumerable will be enumerated when the task is await.

More specifically: With (Fluent) NHibernate you'd typically avoid using .ToList() on queries as otherwise you may end up reading the entire result set. This may be way more data that you want.

answered Oct 08 '21 at 06:30

Manfred

5,320
3
35
29

1

In this case the `ToList` doesn't make much of a difference. The `Task.WhenAll` materializes immediately the supplied `IEnumerable>` to an array anyway ([source code](https://referencesource.microsoft.com/mscorlib/system/threading/tasks/Task.cs.html#b6d43f186de3a867)). – Theodor Zoulias Oct 08 '21 at 11:37
1

@TheodorZoulias Fair enough. However, the implementation may change. - Generally, I avoid using `.ToList()` as it also depends on the LINQ provider in question. While at the moment, this is the one at the link you provided, the same is not necessarily true in all scenarios where you might use `.ToList()`. Definitely worth being aware of the particular implementation of the LINQ provider. Therefore, thank you for your very valuable coment, Theodor! – Manfred Oct 08 '21 at 21:49

How to await a list of tasks asynchronously using LINQ?

6 Answers6

Linked

Related