Concurrent calls to an API, keep successful results and ignore errors

Question

I wrote a method to process concurrently several tenth of API calls (see code below).

Thing is the API's server bears up to about 20 simultaneous calls, but above that, it starts to return 500 errors for some calls.
And when this kind of error is raised, all the tasks are somehow "canceled" or "forgot", not only the calls that returned 500s, because the resulting metadatas collection is empty.
I tried to surround Task.WaitAll with a try/catch, but there is no exception throwed here. No surprise, because TryGetMetadataAsync is all included in a try/catch.

I just want to keep results from successful calls and forget errored calls.
Any idea on what I am missing? Maybe there is an issue with cancellation tokens?

FWIW: api.GetMetadataWithHttpInfoAsync is an Open API Generator client (generated from a Swagger file).

public async Task<IEnumerable<Metadata>> GetMetadataAsync(IEnumerable<string> dataRefs,
    string? userJwt, CancellationToken cancellationToken)
{
    var userId = GetUserIdFromJwt(userJwt);

    var tasks = dataRefs
        .Where(dataRef => !string.IsNullOrEmpty(dataRef))
        .Select(dataRef => TryGetMetadataAsync(dataRef, userId, cancellationToken))
        .ToArray();

    // Load data concurrently and wait for all results:
    Task.WaitAll(tasks, cancellationToken);

    var metadatas = new List<Metadata>();
    foreach (var task in tasks)
    {
        var metadata = await task;
        if (metadata != null) { metadatas.Add(metadata); }
    }

    return metadatas;
}

private async Task<Metadata?> TryGetMetadataAsync(string dataRef, string userId,
    CancellationToken cancellationToken)
{
    try
    {
        if (cancellationToken.IsCancellationRequested) { return null; }

        // Add a custom timeout to 30 seconds
        // (don't wait for potential TaskCanceledException that may be due
        // to 500 errors):
        using var timeoutCancellationTokenSource = new CancellationTokenSource(
            TimeSpan.FromSeconds(30));

        var id = GetIdFromDataRef(dataRef);
        if (id == null) { return null; }

        var metadataResponse = await api.GetMetadataWithHttpInfoAsync(id, userId,
            timeoutCancellationTokenSource.Token);

        if (metadataResponse.StatusCode == HttpStatusCode.OK
            && metadataResponse.Data != null)
        {
            // Just a simple object conversion here...
            return await ConvertToMetadataAsync(dataRef, metadataResponse.Data);
        }
        else
        {
            // Error management removed for brevity...
        }
    }
    catch (ApiException apiEx)
    {
        // Exception specific management removed for brevity...
    }
    catch (Exception ex)
    {
        // Exception management removed for brevity...
    }

    return null;
}

Answer 1

Thanks to @TheodorZoulias' and @JohnWu's comments, the solution was to use the combination of Task.WhenAll and ContinueWith (indeed, possible duplicate with Is it possible to get successful results from a Task.WhenAll when one of the tasks fails?):

Here is a working code (with input split in chunks to preserve API responsiveness):

public new async Task<IEnumerable<Metadata>> GetMetadataAsync(IEnumerable<string> dataRefs, string? userJwt, CancellationToken cancellationToken)
{
    var userId = GetUserIdFromJwt(userJwt);
    var metadatas = new ConcurrentBag<Metadata>();
    // Chunk size to split input sequence:
    const int chunkSize = 13;

    // Split dataRefs in chunks:
    var drChunks = dataRefs.Partition(chunkSize).ToList();
    foreach (var chunk in drChunks)
    {
        var tasks = chunk
            .Where(partDataRef => !string.IsNullOrEmpty(partDataRef))
            .Select(partDataRef => TryGetMetadataAsync(partDataRef, userId, cancellationToken))
            .ToArray();

        try
        {
            // Use a ContinueWith to keep successful results and ignore errors
            await Task.WhenAll(tasks).ContinueWith(t =>
            {
                // Retrieve successful calls results:
                var chunkMetadatas = tasks
                    .Where(t => t.Status == TaskStatus.RanToCompletion)
                    .Where(t => t.Result != null)
                    .Select(t => t.Result)
                    .Cast<Metadata>()
                    .ToList();
                foreach (var metadata in chunkMetadatas) { metadatas.Add(metadata); }

                // Manage faulted or canceled tasks:
                var aggregateExceptions = tasks
                    .Where(t => t.IsFaulted)
                    .Where(t => t.Exception != null)
                    .Select(t => t.Exception) // The Exception is of type AggregateException
                    .Cast<AggregateException>()
                    .ToArray();
                var exceptions = new AggregateException(aggregateExceptions)
                    .Flatten()
                    .InnerExceptions
                    .ToList(); // Flatten the hierarchy of AggregateExceptions
                if (exceptions.Count > 0)
                {
                    // Logging...
                }
                else if (t.IsCanceled)
                {
                    // No exceptions and at least one task was canceled
                    // Logging...
                }
            }, cancellationToken, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
        }
        catch (Exception ex)
        {
            // Logging...
        }
    }

    return metadatas;
}

Answer 2

Try to use await Task.WhenAll instread of Task.WaitAll Like this:

var tasks = dataRefs
    .Where(dataRef => !string.IsNullOrEmpty(dataRef))
    .Select(dataRef => TryGetMetadataAsync(dataRef, userId, cancellationToken))
    .ToArray();

await Task.WhenAll(tasks);

var metadatas = new List<Metadata>();
foreach (var task in tasks)
{
    var metadata = await task;
    if (metadata != null) { metadatas.Add(metadata); }
}

Probably Task.WaitAll just not continue call generated state machine for TryGetMetadataAsync