Timed out when trying to download a file through HttpClient

Question

So, I'm trying to download an exe from GitHub Releases page, for example PowerToys (https://github.com/microsoft/PowerToys/releases/download/v0.70.1/PowerToysSetup-0.70.1-x64.exe) when I use this link in browser, I can see that there is a redirect and file begins to download, but when I use this code in my .NET 6 project, I am getting a timeout. When trying to download for example an image, using the same code, it works.

HttpResponseMessage message = await _httpClient.GetAsync(sourceUri, cancellationToken); // there is a time out exception
message.EnsureSuccessStatusCode();
return await message.Content.ReadAsStreamAsync();

What am I doing wrong? I've tried set explicitly AllowAutoRedirect to true, MaxAutomaticRedirections, Accept header to "*/*", UserAgent, nothing works, why is it?

Thanks.

EDIT: This is the exception chain:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing. 
---> System.TimeoutException: A task was canceled. 
---> System.Threading.Tasks.TaskCanceledException: A task was canceled..

EDIT: Tried SSL bypass, didn't work.

Seems to work https://dotnetfiddle.net/ZnETy3 although I get a Out of Memory because not using `HttpCompletionOption.ResponseHeadersRead` — Charlieface, Jun 12 '23 at 14:21
That file seems to be 250Mb, it might take longer than 10s to download. — Magnus, Jun 13 '23 at 13:07
I know that that file is 250MB I want to retrieve file as a stream to download it sequentialy. — craftersmine, Jun 13 '23 at 15:29

Loel · Answer 1 · 2023-06-13T11:32:51.023

0

It seems to me that you should extend the debugging - the timeout information alone may not be enough. Well, and make sure you are not sending a cancellationToken.

private static HttpClient _httpClient = new(); private static string? sourceUri = @"https://github.com/microsoft/PowerToys/releases/download/v0.70.1/PowerToysSetup-0.70.1-x64.exe";

static async Task Main(string[] args)
{
    try
    {
        HttpResponseMessage message = await _httpClient.GetAsync(sourceUri); 
        message.EnsureSuccessStatusCode();
         var tmp = await message.Content.ReadAsStreamAsync();
    }
    catch (TaskCanceledException e)
    {
        Console.WriteLine(e.StackTrace);
    }

    Console.ReadKey();
}

Try sending a request to a URL without redirection

Also be aware that creating a new instance of the http client each time can exhaust the pools of available ports.

    private HttpClient _httpClient;

    public InitClient()
    {
        var handler = new HttpClientHandler()
        {
            AllowAutoRedirect = false,
            ServerCertificateCustomValidationCallback = (httpRequestMessage, cert, chain, policyErrors) => true,
        };
        _httpClient = new HttpClient(handler);
    }
   
    public async Task<string> TrackAsync(string trackingUrl, CancellationToken ct)
    {
        var response = await _httpClient.GetAsync(trackingUrl, ct);

        return response.StatusCode == HttpStatusCode.Redirect
             ? response.Headers.Location.OriginalString
             : null;
    }

    //IDisposable stuff...

You can still try doing a bypass on the SLL.

            var handler = new HttpClientHandler()
        {
            ServerCertificateCustomValidationCallback = (httpRequestMessage, cert, chain, policyErrors) => true,
        };

edited Jun 13 '23 at 11:32

answered Jun 12 '23 at 14:24

Loel

23
6

I've updated the question, but even when removing cancellation token from method parameters, there is still the same issue – craftersmine Jun 12 '23 at 14:33
If you run the code in a test console application, you also get this message ? - it is difficult to guess what is happening in the code above. – Loel Jun 12 '23 at 14:42
Created blank console app, still getting same exception chain – craftersmine Jun 12 '23 at 14:58
This is what I'm getting in my browser https://imgur.com/a/CIufqmp – craftersmine Jun 12 '23 at 15:02
mby try var httpClient = new HttpClient(new HttpClientHandler { AllowAutoRedirect = false, }); or simply u will suffice for you: var stream = await _httpClient.GetStreamAsync(sourceUri); – Loel Jun 12 '23 at 15:09
I've already tried them, AllowAutoRedirect is already used there, and I've tried to use GetStreamAsync before, it didn't worked for me because I can't get stream length needed for download progress calculation – craftersmine Jun 12 '23 at 15:27
When using AllowAutoRedirect = false, it just returns 302 HTTP status, which doesn't actually contain any file data – craftersmine Jun 12 '23 at 15:34
do you need to download files using httpclient ? using (var client = new WebClient()) { client.DownloadFile("https://github.com/microsoft/PowerToys/releases/download/v0.70.1/PowerToysSetup-0.70.1-x64.exe", "tmp.exe"); } – Loel Jun 12 '23 at 17:43
or try these answers https://stackoverflow.com/a/66270371/21579293 – Loel Jun 12 '23 at 18:09
WebClient is obsolete. Anyway, the issue is at GetAsync, I don't get response message and it times out, rather than giving me some kind of error, while I can successfully request this file in my browser – craftersmine Jun 12 '23 at 18:53
Hi, And how was it, did you manage to find a solution ? Have you reviewed this description of the problem ? https://dev.to/joaofbantunes/using-nets-httpclient-without-following-redirects-21k3 https://github.com/joaofbantunes/HttpClientNotFollowingRedirectsSample – Loel Jun 13 '23 at 07:58
Nope, I didn't managed to find a solution. This links are related to ASP.NET Core, and I'm using a plain console application. – craftersmine Jun 13 '23 at 11:07
I have updated my answer, added bypass on SLL – Loel Jun 13 '23 at 11:34
Nope, SSL bypass doesn't work – craftersmine Jun 13 '23 at 12:19

score 0 · Answer 2 · answered Jun 23 '23 at 11:46

Reading the stream with a buffer works. I am not an expert so I guess the buffering could be improved. However this is working on my machine.

  static async Task Main(string[] args)
        {
            using var httpClient = new HttpClient();

            var fileUrl = "https://github.com/microsoft/PowerToys/releases/download/v0.70.1/PowerToysSetup-0.70.1-x64.exe"; // Replace with the actual file URL

            using var response = await httpClient.GetAsync(fileUrl, HttpCompletionOption.ResponseHeadersRead);
            response.EnsureSuccessStatusCode();

            var fileName = Path.GetFileName(fileUrl);
            var filePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, fileName);

            using var fileStream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);
            using var contentStream = await response.Content.ReadAsStreamAsync();

            var buffer = new byte[4096];
            int bytesRead;
            long totalBytesRead = 0;
            long totalBytes = response.Content.Headers.ContentLength ?? -1;

            while ((bytesRead = await contentStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
            {
                await fileStream.WriteAsync(buffer, 0, bytesRead);
                totalBytesRead += bytesRead;

                if (totalBytes > 0)
                {
                    var progress = (int)((double)totalBytesRead / totalBytes * 100);
                    Console.WriteLine($"Downloading... {progress}%");
                }
            }

            Console.WriteLine("File downloaded successfully.");
            Console.WriteLine("File saved at: " + filePath);
        }

Timed out when trying to download a file through HttpClient

2 Answers2