1

I got a weird situation when using C# HttpClient. I am trying to use the HttpCompletionOption.ResponseHeadersRead option in GetAsync to get response headers without content as quickly as possible. But when downloading files, I am in await GetAsync until the whole content is downloaded over the network (i checked this with Fiddler). I am attaching an example code that downloads a 1Gb test file. The example application will hang in the await client.GetAsync until all file content is received over the network. How do I get control back when the headers have finished receiving and not wait for the complete content transfer over the network?

using System;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

public class Program
    {
        private const int HttpBufferSize = 81920;

        private static async Task Main(string[] args)
        {
            var url = new Uri("http://212.183.159.230/1GB.zip");

            await DownloadFileAsync(@"C:\1GB.zip", url, CancellationToken.None).ConfigureAwait(false);
        }

        private static async Task DownloadFileAsync(string filePath, Uri fileEndpoint,
            CancellationToken token)
        {
            using var client = new HttpClient();

            using var response = await client.GetAsync(fileEndpoint, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false);

            response.EnsureSuccessStatusCode();

            await using var contentStream = await response.Content.ReadAsStreamAsync(token).ConfigureAwait(false);
            await using var stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);
            await contentStream.CopyToAsync(stream, HttpBufferSize, token).ConfigureAwait(false);
        }
    }
Paul Rozhkin
  • 61
  • 1
  • 6
  • Instead of `GET` request you should be using `HEAD`. – Riddell Jun 30 '21 at 10:51
  • @Riddell thats what httpcompletionoption does in the Get call i think https://stackoverflow.com/questions/16416699/http-head-request-with-httpclient-in-net-4-5-and-c-sharp – sommmen Jun 30 '21 at 10:53
  • 1
    @sommmen From memory, `GetAsync` is always a `GET` request as it's a wrapper around `SendAsync` with `HttpMethod.Get`. – Riddell Jun 30 '21 at 10:57
  • I'm unable to reproduce this with the code provided. The code does not wait long on `GetAsync` for me; it waits long on `CopyToAsync`, as expected. Is it possible that Fiddler is misleading you? – Stephen Cleary Jun 30 '21 at 13:04
  • @StephenCleary Unfortunately for me, the code did take a long time for `GetAsync` and fast for `CopyToAsync`. The reason for this was the Fiddler (proxy server), which does not seem to return a partially received response to the sender. See my answer for more details. We now know to be careful with Fiddler... – Paul Rozhkin Jun 30 '21 at 15:02

2 Answers2

2

You are sending a GET request. If you only require the headers then you can use HEAD request. An example for HttpClient:

client.SendAsync(new HttpRequestMessage(HttpMethod.Head, url))

Caution: Servers can block HEAD requests so make sure to handle gracefully. For example, fallback to GET request if the response fails but it will be at the cost of speed.

Riddell
  • 1,429
  • 11
  • 22
  • I need to receive not only headers, so I cannot perform only the Head request. Also, in my task, it is required to perform actions between getting headers and content, so it is not possible to execute Head and then Get. – Paul Rozhkin Jun 30 '21 at 14:03
2

I have identified the reason for this behavior. The reason was Fiddler. It acted as a proxy and did not seem to redirect partially received responses. To check this, I've added console output for each of the operations:

Console.WriteLine($"Start GetAsync - {DateTime.Now}");
using var response = await client.GetAsync(fileEndpoint, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false);
Console.WriteLine($"End GetAsync - {DateTime.Now}");

response.EnsureSuccessStatusCode();

await using var contentStream = await response.Content.ReadAsStreamAsync(token).ConfigureAwait(false);
await using var stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);

Console.WriteLine($"Start CopyToAsync - {DateTime.Now}");
await contentStream.CopyToAsync(stream, HttpBufferSize, token).ConfigureAwait(false);
Console.WriteLine($"End CopyToAsync - {DateTime.Now}");

Results with running program Fiddler:

Start GetAsync - 30.06.2021 17:46:03
End GetAsync - 30.06.2021 17:46:49
Start CopyToAsync - 30.06.2021 17:46:49
End CopyToAsync - 30.06.2021 17:46:51

Results without Fiddler:

Start GetAsync - 30.06.2021 17:38:32
End GetAsync - 30.06.2021 17:38:32
Start CopyToAsync - 30.06.2021 17:38:32
End CopyToAsync - 30.06.2021 17:39:48

Conclusion: be careful with proxies

Paul Rozhkin
  • 61
  • 1
  • 6