Keep running specific number of task async

Question

I'm currently working on a concurrent file downloader.

For that reason I want to parametrize the number of concurrent tasks. I don't want to wait for all the tasks to be completed but to keep the same number being runned.

In fact, this thread on star overflow gave me a proper clue, but I'm struggling making it async:

Keep running a specific number of tasks

Here is my code:

public async Task StartAsync()
    {
        var semaphore = new SemaphoreSlim(1, _concurrentTransfers);
        var queueHasMessages = true;

        while (queueHasMessages)
        {
            try {
                await Task.Run(async () =>
                  {
                      await semaphore.WaitAsync();
                      await asyncStuff();
                 });
            }
            finally {
                semaphore.Release();
            };
        }
    }

But the code just get executed one at a time. I think that the await is blocking me for generating the desired amount of tasks, but I don't know how to avoid it while respecting the limit established by the semaphore.

If I add all the tasks to a list and make a whenall, the semaphore throws an exception since it has reached the max count.

Any suggestions?

Answer 1

It was brought to my attention that the struck-through solution will drop any exceptions that occur during execution. That's bad.

Here is a solution that will not drop exceptions:

---

Task.Run is a Factory Method for creating a Task. You can check yourself with the intellisense return value. You can assign the returned Task anywhere you like.

"await" is an operator that will wait until the task it operates on completes. You are able to use any Task with the await operator.

public static async Task RunTasksConcurrently()
{
    IList<Task> tasks = new List<Task>();

    for (int i = 1; i < 4; i++)
    {
        tasks.Add(RunNextTask());
    }

    foreach (var task in tasks) {
        await task; 
    }
}

public static async Task RunNextTask()
{
    while(true) {
        await Task.Delay(500);
    }
}

By adding the values of the Task we create to a list, we can await them later on in execution.

---

Previous Answer below

Edit: With the clarification I think I understand better.

Instead of running every task at once, you want to start 3 tasks, and as soon as a task is finished, run the next one.

I believe this can happen using the .ContinueWith(Action<Task>) method.

See if this gets closer to your intended solution.

    public void SpawnInitialTasks()
    {
        for (int i = 0; i < 3; i++)
        {
            RunNextTask();
        }
    }

    public void RunNextTask()
    {
        Task.Run(async () => await Task.Delay(500))
            .ContinueWith(t => RunNextTask());  
        // Recurse here to keep running tasks whenever we finish one.
    }

The idea is that we spawn 3 tasks right away, then whenever one finishes we spawn the next. If you need to keep data flowing between the tasks, you can use parameters:

RunNextTask(DataObject object)

Answer 2

You can do this easily the old-fashioned way without using await by using Parallel.ForEach(), which lets you specify the maximum number of concurrent threads to use.

For example:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

namespace Demo
{
    class Program
    {
        public static void Main(string[] args)
        {
            IEnumerable<string> filenames = Enumerable.Range(1, 100).Select(x => x.ToString());

            Parallel.ForEach(
                filenames,
                new ParallelOptions { MaxDegreeOfParallelism = 4},
                download
            );
        }

        static void download(string filepath)
        {
            Console.WriteLine("Downloading " + filepath);
            Thread.Sleep(1000); // Simulate downloading time.
            Console.WriteLine("Downloaded " + filepath);
        }
    }
}

If you run this and observe the output, you'll see that the "files" are being "downloaded" in batchs.

A better simulation is the change download() so that it takes a random amount of time to process each "file", like so:

static Random rng = new Random();

static void download(string filepath)
{
    Console.WriteLine("Downloading " + filepath);
    Thread.Sleep(500 + rng.Next(1000)); // Simulate random downloading time.
    Console.WriteLine("Downloaded " + filepath);
}

Try that and see the difference in the output.

---

However, if you want a more modern way to do this, you could look into the Dataflow part of the TPL (Task Parallel Library) - this works well with async methods.

This is a lot more complicated to get to grips with, but it's a lot more powerful. You could use an ActionBlock to do it, but describing how to do that is a bit beyond the scope of an answer I could give here.

Have a look at this other answer on StackOverflow; it gives a brief example.

Also note that the TPL is not built in to .Net - you have to get it from NuGet.