3

If I have a list of tasks which I want to execute together but at the same time I want to execute a certain number of them together, so I await for one of them until one of them finishes, which then should mean awaiting should stop and a new task should be allowed to start, but when one of them finishes, I don't know how to stop awaiting for the task which is currently being awaited, I don't want to cancel the task, just stop awaiting and let it continue running in the background.

I have the following code

foreach (var link in SharedVars.DownloadQueue)
{
    if (currentRunningCount != batch)
    {
        var task = DownloadFile(extraPathPerLink, link, totalLen);
        _ = task.ContinueWith(_ =>
        {
            downloadQueueLinksTasks.Remove(task);
            currentRunningCount--;
            // TODO SHOULD CHANGE WHAT IS AWAITED
        });
        currentRunningCount++;
        downloadQueueLinksTasks.Add(task);
    }

    if (currentRunningCount == batch)
    {
        // TODO SHOULD NOT AWAIT 0
        await downloadQueueLinksTasks[0];
    }
}

I found about Task.WhenAny but from this comment here I understood that the other tasks will be ignored so it's not the solution I want to achieve.

I'm sorry if the question is stupid or wrong but I can't seem to find any information related on how to solve it, or what is the name of the operation I want to achieve so I can even search correctly.

Solution Edit

All the answers provided are correct, I accepted the one I decided to use but still all of them are correct.

Thank you everyone, I learned a lot from all of you from these different answers and different ways to approach the problem and how to think about it.

What I learned about this specific problem was that I still needed to await for the other tasks left, so the solution was to have the Task.WhenAny inside the loop (which returns the finished task (this is also important)) AND Task.WhenAll outside the loop to await the other left tasks.

Anton Kahwaji
  • 467
  • 4
  • 15

4 Answers4

2

You can use Task.WaitAny()

Here is the demonstration of the behavior:

public static async Task Main(string[] args)
{
     IList<Task> tasks = new List<Task>();

    tasks.Add(TestAsync(0));
    tasks.Add(TestAsync(1));
    tasks.Add(TestAsync(2));
    tasks.Add(TestAsync(3));
    tasks.Add(TestAsync(4));
    tasks.Add(TestAsync(5));

    var result = Task.WaitAny(tasks.ToArray());

    Console.WriteLine("returned task id is {0}", result);

    ///do other operations where

    //before exiting wait for other tasks so that your tasks won't get cancellation signal
    await Task.WhenAll(tasks.ToArray());

}

public static async Task TestAsync(int i)
{
    Console.WriteLine("Staring to wait" + i);
    await Task.Delay(new Random().Next(1000, 10000));
    Console.WriteLine("Task finished" + i);
}

Output:

Staring to wait0
Staring to wait1
Staring to wait2
Staring to wait3
Staring to wait4
Staring to wait5
Task finished0
returned task id is 0
Task finished4
Task finished2
Task finished1
Task finished5
Task finished3
Derviş Kayımbaşıoğlu
  • 28,492
  • 4
  • 50
  • 72
  • When I try using that for some reason after the first batch executes, the for loop exits, but when I try to use a debugger line by line it works as expected (I can press continue after a few lines and it'll continue working perfectly). – Anton Kahwaji Feb 26 '19 at 21:26
  • 1
    because you are terminating your application. eventually you need to await your tasks. . check my edited answer please – Derviş Kayımbaşıoğlu Feb 26 '19 at 23:00
  • Oh sorry, I didn't notice that line, I didn't include it at all, I thought the inner Task.WhenAny will do the job but forgot that some tasks will not reach it because of the `if`s I had. – Anton Kahwaji Feb 27 '19 at 00:24
2

Task.WhenAny returns the Task which completed.

foreach (var link in SharedVars.DownloadQueue)
{
    var task = DownloadFile(extraPathPerLink, link, totalLen);
    downloadQueueLinksTasks.Add(task);

    if (downloadQueueLinksTasks.Count == batch)
    {
        // Wait for any Task to complete, then remove it from
        // the list of pending tasks.
        var completedTask = await Task.WhenAny(downloadQueueLinksTasks);
        downloadQueueLinksTasks.Remove(completedTask);
    }
}

// Wait for all of the remaining Tasks to complete
await Task.WhenAll(downloadQueueLinksTasks);
Daniel Gray
  • 1,697
  • 1
  • 21
  • 41
canton7
  • 37,633
  • 3
  • 64
  • 77
  • But this won't let the other tasks execute in batch as well, for example if batch is set to 5, I want to have at max 5 tasks executing at once but once one of them finishes, another one should start but not all the others (the others should wait in the same way) – Anton Kahwaji Feb 26 '19 at 21:31
  • 1
    Yes, that is what this code does. The Tasks are executing whether they're being awaited or not. This code starts 5 Tasks, then waits until one of them finishes. When one finishes, it removes it from `downloadQueueLinksTasks`, starts another Task, and waits for the next one to finish. Try it out and see. – canton7 Feb 26 '19 at 21:39
  • 1
    Oh my god, the solution works, it works exactly as I planned, thank you very much. – Anton Kahwaji Feb 26 '19 at 21:43
2

What you're asking about is throttling, which for asynchronous code is best expressed via SemaphoreSlim:

var semaphore = new SemaphoreSlim(batch);
var tasks = SharedVars.DownloadQueue.Select(link =>
{
  await semaphore.WaitAsync();
  try { return DownloadFile(extraPathPerLink, link, totalLen); }
  finally { semaphore.Release(); }
});
var results = await Task.WhenAll(tasks);
Stephen Cleary
  • 437,863
  • 77
  • 675
  • 810
1

You should use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:

IDisposable subscription =
    SharedVars.DownloadQueue
        .ToObservable()
        .Select(link =>
            Observable.FromAsync(() => DownloadFile(extraPathPerLink, link, totalLen)))
        .Merge(batch) //max concurrent downloads
        .Subscribe(file =>
        {
            /* process downloaded file here
               (no longer a task) */
        });

If you need to stop the downloads before they would naturally finish just call subscription.Dispose().

Enigmativity
  • 113,464
  • 11
  • 89
  • 172
  • This looks promising, but I have one question since I can't find any information regarding it, is Rx cross platform? Since it depends on .Net Framework and not on .Net Core I'm confused. – Anton Kahwaji Feb 26 '19 at 21:39