How after completing any task in the loop, pass a value to another method, return to the loop, and wait for the next task to complete?

Question

There are 2 methods, one method depends on the other. I need to send several files to the server in parallel, which answer from the server comes first, and send it to DownloadFileAsync method, and do the same with the rest (without waiting for all answers at once, send them (answers from the server) to DownloadFileAsync() method, as they are received).

public async Task CompressAllFilesAsync(List<UserFile> files, string outputpath)
{
    await UploadAllFilesAsync(files);
    await DownloadAllFilesAsync(files, outputpath);
}

public async Task UploadAllFilesAsync(List<UserFile> files)
{
    IEnumerable<Task> allTasks = files.Select(async (file, i) =>
        files[i].FileLink = await UploadFileAsync(files[i])
    );

    await Task.WhenAll(allTasks);
}

public async Task DownloadAllFilesAsync(List<UserFile> files, string outputpath)
{
    IEnumerable<Task> allTasks = files.Select((file, i) =>
        DownloadFileAsync(files[i].FileLink,
        Path.Combine(outputpath, $"{files[i].FileName}")
    ));

    await Task.WhenAll(allTasks);
}

Now the program is waiting for all answers from the server (links to downloadable files) before running the DownloadAllFilesAsync() method.

Image-Example

Theodor Zoulias · Accepted Answer · 2022-02-13T13:47:37.080

1

You could consider using the new (.NET 6) API Parallel.ForEachAsync, and execute the UploadFileAsync and then the DownloadFileAsync sequentially for each UserFile, but concurrently with other files:

public async Task CompressAllFilesAsync(List<UserFile> files, string outputPath)
{
    var options = new ParallelOptions() { MaxDegreeOfParallelism = 10 };
    await Parallel.ForEachAsync(files, options, async (file, ct) =>
    {
        await UploadFileAsync(file);
        await DownloadFileAsync(file.FileLink,
            Path.Combine(outputPath, $"{file.FileName}"));
    });
}

The UploadFileAsync and DownloadFileAsync will be invoked on ThreadPool threads. This shouldn't be a problem, unless those methods interact with thread-affine components (like UI controls).

edited Feb 13 '22 at 13:47

answered Feb 13 '22 at 13:19

Theodor Zoulias

34,835
7
69
104

What is "ct" inside "async (file, ct)"? – Ilya Feb 13 '22 at 13:26
@Ilya it's a `CancellationToken`, that you could use to cancel the processing of one file, if another file failed (for faster propagation of the error). It seems that your `UploadFileAsync` and `DownloadFileAsync` methods are not cancelable, so you can't pass this token to them, but you might consider intercepting a `ct.ThrowIfCancellationRequested()` between the two calls. – Theodor Zoulias Feb 13 '22 at 13:39
I can't seem to use "Parallel.ForEachAsync", in Visual Studio 2017 with "Target Platform 4.6.1", this method is not available. – Ilya Feb 13 '22 at 14:14
@Ilya can you download the Visual Studio 2022 and target the newest .NET 6 runtime? If not, then you can't use the `Parallel.ForEachAsync` API. You'll have to find an alternative to this method (there are many custom implementations available), like the one that I've posted [here](https://stackoverflow.com/questions/11564506/nesting-await-in-parallel-foreach/65251949#65251949). – Theodor Zoulias Feb 13 '22 at 14:18
Thank you for your help with the code, I managed to find another solution that does the same as your code and does not require (.NET 5 | 6). – Ilya Feb 16 '22 at 10:02
@Ilya you're welcome! I preferred to show a solution that targets the latest .NET runtime, because it will be applicable to a larger audience in the long run. Using instead the `Task.WhenAll` and a `SemaphoreSlim` is a valid approach. It's a bit clumsy compared to the elegant `Parallel.ForEachAsync`, but it works. – Theodor Zoulias Feb 16 '22 at 10:07

Ilya · Answer 2 · 2022-03-08T09:39:51.310

public async Task CompressAllFilesAsync(List<UserFile> files, string outputpath) {
    SemaphoreSlim throttler = new SemaphoreSlim(2);

    IEnumerable<Task> allTasks = files.Select(async (file) => {
        await throttler.WaitAsync();

        try {
            file.Link = await UploadFileAsync(file);

            await DownloadFileAsync(file.Link,
                Path.Combine(outputpath, file.Name));
        } finally {
            throttler.Release();
        }
    });

    await Task.WhenAll(allTasks);
}

score -2 · Answer 3 · answered Feb 12 '22 at 16:38

-2

for (int i = 0; i < fileToSend; i++)
{    
   Task task = new Task(() => 
   {
      SendFile(file[i]);
   });
   task.Start();
}

This way you're gonna create a background task that will send your files in paralell

answered Feb 12 '22 at 16:38

Necrolytie

21
3

This solution didn't work, the DownloadFile() method gets the wrong data. – Ilya Feb 13 '22 at 07:43
This code is roughly the same as `Task.Run()` without actually awaiting for the operation to complete. There's seldom any good reason for cold task that are explicitly started with `Start()`. A Task isn't a thread, it's a promise. Calling `Start()` will do the same thing as `Task.Run()`: schedule the promise for execution by a threadpool thread – Panagiotis Kanavos Feb 15 '22 at 15:36

How after completing any task in the loop, pass a value to another method, return to the loop, and wait for the next task to complete?

3 Answers3