Running Multiple Tasks parallely results in System.AggregateException

Question

I am fetching data from an API with different dataset-identifiers as a query parameters. Earlier I tried a synchronous approach but its taking too much time.So I decided to create Tasks dynamically inside a foreach loop for every dataset-identifier. Here is the code that I have written

foreach (var dataset in datasets)
{
    Task task = new Task(() =>
    {
        using (var client = new HttpClient())
        {
            var urlParameters = $"?apikey={apiKey}&dataset={dataset}";
            client.BaseAddress = new Uri(url);
            HttpResponseMessage response = client.GetAsync(urlParameters).Result;
            var yearlyDatasetResult = response.Content.ReadAsStringAsync().Result;
            var datasetCsvHeader = yearlyDatasetResult.Split(new[] { '\r', '\n' })
                .FirstOrDefault();
            var headers = datasetCsvHeader.Split(';');
            if (csvHeaders.Length < headers.Length)
                csvHeaders = headers;

            var content = yearlyDatasetResult.Substring(datasetCsvHeader.Length);
            using (var file = new StreamWriter("C:\\xxxx\\xxxx\\source\\"
                + $"repos\\xxxx\\CSV\\{dataset}.csv"))
            {
                file.WriteLine(content);
            }
        }
    });

    task.Start();
    tasksToWait.Add(task);
}

Task.WaitAll(tasksToWait.ToArray());
Console.WriteLine("All tasks completed");

My goal is to fetch all the CSV files and then write to my system without the headers via parallel running tasks, but I am getting System.AggregateException. Could it be because of the blocking nature of this line?

HttpResponseMessage response = client.GetAsync(urlParameters).Result;

I am not able to guess the reason for this kind of exception, I do know that one of the tasks is failing and that's why I am getting this aggregated exception and that's the nature of Task.WaitAll(...), What could be the possible solution to this ?

An AggregateException should contain one or more other exceptions that tell you exactly what the issue is, but yeah, I'd guess it's the blocking calls and the fact that you don't even `await` the `Task.WaitAll`. You should really commit to using async/await. — juharr, May 12 '20 at 11:42
the exception is happening during the execution of this line? `HttpResponseMessage response = client.GetAsync(urlParameters).Result;` — Richárd Baldauf, May 12 '20 at 11:43
@ShubhamTiwari ah okay. To figure this out I'm suggesting Christopher answer below. Using the async/await feautre would be usefull to identify the source of the exception ^.^ — Richárd Baldauf, May 12 '20 at 12:11

score 3 · Answer 1 · answered May 12 '20 at 11:44

Running Multiple Tasks parallely results in System.AggregateException

That is a slightly incorrect assesment. Your code is throwing Exceptions. AggregateException is only there so you do not ignore them by accident.

Multtiasking is notoriously good at swallowing Exceptions. And as that is the only way to communicate them, each task in a Execution line might throw Exceptions and Exceptions should never be ignored, all Exceptions are collected and raised wrapped neatly into a AggregateException.

You goal is to figure out what those Exceptions are, to properly classify them and then properly handle them. Without the inner exceptions it is hard to say

score 2 · Answer 2 · answered May 12 '20 at 13:38

An AggregateException is just a bag that contains other exceptions. You can access these exceptions through the property InnerExceptions:

try
{
    Task.WaitAll(tasksToWait.ToArray());
    Console.WriteLine("All tasks completed");
}
catch (AggregateException aex)
{
    foreach (var ex in aex.Flatten().InnerExceptions)
    {
        Console.WriteLine(ex);
    }
}

In the above example I used the method Flatten just in case there are nested AggregateExceptions, to flatten the tree.

Running Multiple Tasks parallely results in System.AggregateException

2 Answers2