Awaiting tasks from multiple objects

Question

I have an application that uses MEF to load plugins. All of these plugins conform to the following interface:

public interface IPlugin {
    Task Start();
}

All the methods are implemented as async: public async Task Start()

When the application is running I have an IEnumerable<IPlugin> property available with all the plugins. The question is basically how I can run all the Start() methods in parallel and wait until all methods are finished?

I know about Parallel.ForEach(plugins, plugin => plugin.Start()), but this is not awaitable and execution continues before all plugins are started.

The most promising solution seems to be Task.WhenAll(), but I don't know how to send in an unknown list of methods into this without adding some scaffolding (which seems like overhead).

How can I accomplish this?

Sergey.quixoticaxis.Ivanov · Accepted Answer · 2018-09-01T09:52:26.707

5

And here's a one-liner:

await Task.WhenAll(plugins.Select(p => p.Start()));

The plugins will run asynchronously, but not in parallel. If you want for some reason to dispatch plugins to thread pool explicitly, you may add Task.Run with async lambda into Select.

edited Sep 01 '18 at 09:52

answered Sep 01 '18 at 08:34

Sergey.quixoticaxis.Ivanov

1,387
12
22

2

Son of a bench, it's been staring me in the face this whole time! Would you care to elaborate on "The plugins will run asynchronously, but not in parallel" ? – TheHvidsten Sep 01 '18 at 09:43
You can output `CurrentThread`'s id and you will see that plugins at least start on a single thread. Here's a nice article about it: https://blogs.msdn.microsoft.com/benwilli/2015/09/10/tasks-are-still-not-threads-and-async-is-not-parallel/ – Sergey.quixoticaxis.Ivanov Sep 01 '18 at 09:51
@Sergey.quixoticaxis.Ivanov For running in parallel, might it not be better to use `AsParallel()` (as in `plugins.AsParallel().Select(p => p.Start())`) instead of `Task.Run`? See [here](https://stackoverflow.com/questions/5009181/parallel-foreach-vs-task-factory-startnew). – pere57 Sep 01 '18 at 10:50
1

@pere57 I doubt it will make any difference. Also I'm not certain that plugins can get any value out of work "chunking" inside parallel query. It should be measured irl, what is better for this case. – Sergey.quixoticaxis.Ivanov Sep 01 '18 at 11:01
*"The plugins will run asynchronously, but not in parallel."* -- This is not correct. Assuming that the `Start` method returns an incomplete task, in other words that it is truly asynchronous, which is a reasonable assumption to make, the plugins will run concurrently. Multiple asynchronous operations will be in-flight at the same time. – Theodor Zoulias Feb 13 '23 at 04:40
@TheodorZoulias, it depends on what you consider "parallel", and, in general, it depends. If there's a single-threaded context, the plugins would run on a single thread interchanged. Even in the answer I didn't mention how to run plugins in parallel (only how to offload them to thread pool explicitly), because nor thread pool usage, nor even manually creating a separate thread for each plugin give any guarantees of parallel execution. – Sergey.quixoticaxis.Ivanov Feb 15 '23 at 16:24
Sergey what are the assumptions that you've made about the `Task Start();` method? We have no idea about how this method is implemented in the concrete `IPlugin` types. It might be `Task Start() => Task.Factory.StartNew(CpuBoundAction, TaskCreationOptions.LongRunning);`, in which case your statement *"The plugins will not run in parallel."* doesn't hold. The action of each task will be running on a dedicated thread, in parallel with other task's actions. Assuming of course that the program runs on a machine with multiple cores. – Theodor Zoulias Feb 15 '23 at 17:05
@TheodorZoulias, none assumptions whatsoever. The OP asked the question of how to await `IEnumerable`, and also mentioned the goal of running `Start()` (whatever inside) in parallel. I gave the answer to the first question, mentioning that the second is not guaranteed. If you want I can re-phrase it as "asynchronously, but not necessarily in parallel". Off-topic: `LongRunning` task also does not guarantee parallel execution, AFAIR, it simply notifies `ThreadPool` that may choose to not spawn a thread (sorry, I don't want to dive into the sources right now). – Sergey.quixoticaxis.Ivanov Feb 16 '23 at 20:42

Frank Fajardo · Answer 2 · 2018-09-01T08:25:01.073

3

You can do:

var tasks = new List<Task>();
foreach(var plugin in plugins) 
{
   var task = plugin.Start();
   tasks.Add(task);
}
await Task.WhenAll(tasks);

edited Sep 01 '18 at 08:25

answered Sep 01 '18 at 08:13

Frank Fajardo

7,034
1
29
47

2

My pet peeve is `if(collection.Count > 0)` - use `if(collection.Any())`. You don't care how many items, just 'is there any?'. – Neil Sep 01 '18 at 08:16
@Neil, very good suggestion! I always keep forgetting that `Any()` is more efficient in this case. :) – Frank Fajardo Sep 01 '18 at 08:16
Or you can ommit `Any` check as `WhenAll` can work with zero tasks. – Sergey.quixoticaxis.Ivanov Sep 01 '18 at 08:21
Also you probably meant `Task.WhenAll`, not `Tasks.WhenAll` – Sergey.quixoticaxis.Ivanov Sep 01 '18 at 08:22
2

@Sergey.quixoticaxis.Ivanov, I didn't know you could pass an empty collection to it, so thanks for the suggestion. – Frank Fajardo Sep 01 '18 at 08:27

score 2 · Answer 3 · answered Sep 01 '18 at 09:51

2

You could use Microsoft's Reactive Framework to ensure that this is awaitable, happens asynchronously and in parallel.

await
    plugins
        .ToObservable()
        .SelectMany(plugin => Observable.FromAsync(() => plugin.Start()))
        .ToArray();

answered Sep 01 '18 at 09:51

Enigmativity

113,464
11
89
172

score 0 · Answer 4 · answered Sep 01 '18 at 08:29

As you can see Start method returns a Task. I would define a list of plugin loading tasks and check with Task.WhenAll when every task is completed. After that you can assume all Start methods have returned.

List<IPlugin> plugins = ... 
var pluginsLoadingTasks = new List<Task>();

foreach(var plugin in plugins)
{
    pluginsLoadingTasks.Add(plugin.Start());
}

// It's not necessary to check if pluginsLoadingTasks is empty, 
// because WhenAll won't throw an exception in that case
await Task.WhenAll(pluginsLoadingTasks);
// You can assume all Start methods have completed

I suggest you to read the differences between Task.WhenAll and Parallel.ForEach constructs.

How is your answer different from what Frank suggested? – Dmytro Mukalov Sep 01 '18 at 08:39 — Dmytro Mukalov, Sep 01 '18 at 08:39

Awaiting tasks from multiple objects

4 Answers4