How to add timeout and retry to url call in C#?

Question

I have a .tgz file that I need to download given a url inside a Testing folder. I am able to download the .tgz file successfully from the url using WebClient.

Below is my code:

private void DownloadTGZFile(string url, string fileToDownload)
{
    using (var client = new WebClient())
    {
        client.DownloadFile(url + fileToDownload, "Testing/configs.tgz");
    }
}

I wanted to see on how can I add a timeout to this call so that if url doesn't respond back within a particular time then it should timeout but it can retry for 3 times and then give up. Also I wanted to see on how can I use HttpClient here instead of WebClient considering it is an older BCL class and not recommended.

Answer 1

To download a file with HttpClient you can do:

// Is better to not initialize a new HttpClient each time you make a request, 
// it could cause socket exhaustion
private static HttpClient _httpClient = new HttpClient()
{
    Timeout = TimeSpan.FromSeconds(5)
};

public async Task<byte[]> GetFile(string fileUrl)
{
    using (var httpResponse = await _httpClient.GetAsync(fileUrl))
    {
        // Throws an exception if response status code isn't 200
        httpResponse.EnsureSuccessStatusCode();
        return await httpResponse.Content.ReadAsByteArrayAsync();
    }
}

For more details about socket exhaustion with HttpClient

As you see, to define a timeout for the Http call you should set a timeout while creating a new HttpClient.

---

To implement a retry policy for the previous code, I would install Polly NuGet package and then:

public async Task<byte[]> GetFile(string fileUrl)
{
    return await Policy
       .Handle<TaskCanceledException>() // The exception thrown by HttpClient when goes in timeout
       .WaitAndRetryAsync(retryCount: 3, sleepDurationProvider: i => TimeSpan.FromMilliseconds(300))
       .ExecuteAsync(async () =>
       {
           using (var httpResponse = await _httpClient.GetAsync(fileUrl))
           {
               // Throws an exception if response status code isn't 200
               httpResponse.EnsureSuccessStatusCode();
               return await httpResponse.Content.ReadAsByteArrayAsync();
           }
       });
}

In this case I defined a retry of 3 times with an interval of 300 milliseconds between each tentative. Also note that I didn't defined the retry for every kind of Exception, because if - for example - you put an invalid URL, retrying is nonsense.

At the end, if you want to save that byte array to a file, you can just do:

File.WriteAllBytes(@"MyPath\file.extension", byteArray);

Answer 2

You can use this function with no dependencies to external libraries. It works for any file size.

EDIT Version to progapate the TaskCanceledException.

public async Task<bool> DownloadFileAsync(string url,
    string destinationFile,
    TimeSpan timeout,
    int maxTries = 3,
    CancellationToken token = default)
{
    using (var client = new HttpClient { Timeout = timeout })
    {
        for (var i = 0; i < maxTries; i++, token.ThrowIfCancellationRequested())
        {
            try
            {
                var response = await client.GetAsync(url, token);
                if (!response.IsSuccessStatusCode)
                    continue;

                var responseStream = await response.Content.ReadAsStreamAsync();
                using (var outputStream = new FileStream(destinationFile, FileMode.Create, FileAccess.Write))
                {
                    await responseStream.CopyToAsync(outputStream, 8 * 1024, token);
                    return true;
                }
            }
            catch (HttpRequestException)
            {
                //ignore
            }
        }
        return false;
    }
}