How to retrieve partial response with System.Net.HttpClient

Question

I'm trying to use the new HttpClient class (in .NET 4.5) to retrieve partial responses from the server in order to check the content. I need to limit the size of data retrieved to the first few bytes of content in the HTTP requests to limit the bandwidth usage.

I've been unable to accomplish this. I have tried using GetAsync(url, HttpCompletionOption.ResponseHeadersRead) then use Content.ReadAsStream() in an attempt to only read the headers and then read the response stream in a small chunk. I also tried GetStreamAsync() and then reading the Content stream with a small chunk (1000 bytes).

In both cases it appears that HttpClient is pulling and buffering the entire HTTP response rather than just reading the requested byte count from the stream.

Initially I was using Fiddler to monitor the data, but realized that Fiddler might actually be causing the entire content to be proxied. I switched to using System.Net tracing (which shows):

ConnectStream#6044116::ConnectStream(Buffered 16712 bytes.)

which is the full size rather than just the 1000 bytes read. I've also double checked in Wireshark to verify that indeed the the full content is being pulled over the wire and it is. With larger content (like a 110k link) I get about 20k of data before the TCP/IP stream is truncated.

The two ways I've tried to read the data:

response = await client.GetAsync(site.Url, HttpCompletionOption.ResponseHeadersRead);
var stream = await response.Content.ReadAsStreamAsync();

var buffer = new byte[1000];                                        
var count = await stream.ReadAsync(buffer, 0, buffer.Length);
response.Close()  // close ASAP
result.LastResponse = Encoding.UTF8.GetString(buffer);

and:

var stream = await client.GetStreamAsync(site.Url);
var buffer = new byte[1000];
var count = await stream.ReadAsync(buffer, 0, buffer.Length);
result.LastResponse = Encoding.UTF8.GetString(buffer);

Both of them produce nearly identical .NET trace's which include the buffered read.

Is it possible to have HttpClient actually read only a small chunk of an Http Repsonse, rather than the entire response in order to not use the full bandwidth? IOW is there a way to disable any buffering on the HTTP connection using either HttpClient or HttpWebRequest?

Update: After some more extensive testing it looks like both HttpClient and HttpWebRequest buffer the first few TCP/IP frames - presumably to ensure the HTTP header is captured. So if you return a small enough request, it tends to get loaded completely just because it's in that inital bufferred read. But when loading a larger content url, the content does get truncated. For HttpClient it's around 20k, for HttpWebRequest somewhere around 8k for me.

Using TcpClient doesn't have any buffering issues. When using it I get content read at the size of the read plus a bit extra for the nearest buffer size overlap, but that does include the HTTP header. Using TcpClient is not really an option for me as we have to deal with SSL, Redirects, Auth, Chunked content etc. At that point I'd be looking at implementing a full custom HTTP client just to turn of buffering.

Answer 1

The best way to achive what you need to do is something like the following:

using System;
using System.Net.Sockets;

namespace tcpclienttest
{
  class Program
  {
    static byte[] GetData(string server, string pageName, int byteCount, out int     actualByteCountRecieved)
    {
      const int port = 80;
      TcpClient client = new TcpClient(server, port);

      string fullRequest = "GET " + pageName + " HTTP/1.1\nHost: " + server + "\n\n";
      byte[] outputData = System.Text.Encoding.ASCII.GetBytes(fullRequest);

      NetworkStream stream = client.GetStream();
      stream.Write(outputData, 0, outputData.Length);

      byte[] inputData = new Byte[byteCount];

      actualByteCountRecieved = stream.Read(inputData, 0, byteCount);

      // If you want the data as a string, set the function return type to a string
      // return 'responseData' rather than 'inputData'
      // and uncomment the next 2 lines
      //string responseData = String.Empty;
      //responseData = System.Text.Encoding.ASCII.GetString(inputData, 0, actualByteCountRecieved);

      stream.Close();
      client.Close();

      return inputData;
    }

    static void Main(string[] args)
    {
      int actualCount;
      const int requestedCount = 1024;
      const string server = "myserver.mydomain.com"; // NOTE: NO Http:// or https:// bit, just domain or IP
      const string page = "/folder/page.ext";

      byte[] myPartialPage = GetData(server, page, requestedCount, out actualCount);
    }
  }
}

Couple of points to note however:

There's NO error handling in there, so you might want to wrap it all in a try/catch or something to make sure you get hold of any connection errors, timeouts, unsolved IP resolution etc.

Beacuse your dealing with the raw stream, then the HTTP headers are also in there, so you'll need to take them into account.

You could in theory, put a loop in just before the main socket read, in keep grabbing data until you get a blank \n on it's own in a line, that will tell you where the headers end, then you could grab your actual count of data, but since I don't know the server your talking too I left that bit out :-)

If you copy/Paste the entire code into a new console project in VS it's runnable as it is, so you can single step it.

As far as I know the HTTP client doesn't make it's raw stream available to the user, and even then if it did because it's allocated as a streaming connection it's not likely you would have much control over it's count, I've looked into it before and given up.

I've used this code a number of times and it works well for me in similar cases, in fact I have a monitor that sits and gets stats from my WiFi adapter using it so I can see who's connecting.

Any questions, feel free to hit me up on here, or ping me on twitter my handle is @shawty_ds (just in case you lost it)

Shawty

Answer 2

I may be wrong at this but I think you're getting confused : when you send the request to the server, it will send you the complete answer through the network. Then it is buffered somewhere by the framework and you access it using the stream. If you don't want the remote server to send you the full answer, you may be able to specify the range of bytes you want using http headers. See HTTP Status: 206 Partial Content and Range Requests for example.

Answer 3

I think/hope this may help.

How can I perform a GET request without downloading the content?

As I suspected, the underlying layers will pull more than you want despite what .NET sees.

Update

Although HttpWebRequest.AddRange(-256) will get the first 256 bytes, it seems that will only work for a static file on IIS.

It sets the Range header (not to be confused with If-Range).

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.2

The server advertises it supports range requests using the Accept-Ranges header.

For Rick's problem this may be good or bad, depending on whether he needs to read static content or not. My guess is that its not what he wants.

An alternative might be to set the ReceiveBufferSize on the ServicePoint which is exposed on WebRequest.

Answer 4

Here's my setup. I don't know why you are seeing the response buffered. Could it be related to the host?

 class Program
    {
        static void Main(string[] args)
        {
            var host = String.Format("http://{0}:8080/", Environment.MachineName);
            var server = CreateServer(host);

            TestBigDownload(host);

            Console.WriteLine("Done");
            server.Dispose();
        }


        private static void TestBigDownload(string host)
        {
            var httpclient = new HttpClient() { BaseAddress = new Uri(host) };
            var stream = httpclient.GetStreamAsync("bigresource").Result;

            var bytes = new byte[10000];
            var bytesread = stream.Read(bytes, 0, 1000);
        }

        private static IDisposable CreateServer(string host)
        {
            var server = WebApp.Start(host, app =>
            {
                var config = new HttpConfiguration();
                config.MapHttpAttributeRoutes();
                app.UseWebApi(config);
            });
            return server;
        }
    }




    [Route("bigresource")]
    public class BigResourceController : ApiController
    {
        public HttpResponseMessage Get()
        {
            var sb = new StringBuilder();
            for (int i = 0; i < 10000; i++)
            {
                sb.Append(i.ToString());
                sb.Append(",");
            }
            var content = new StringContent(sb.ToString());
            var response = new HttpResponseMessage()
            {
                Content = content
            };

            return response;
        }
    }

Logging config

  <system.diagnostics>
    <sources>
      <source name="System.Net">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="System.Net" value="Verbose"/>
    </switches>
    <sharedListeners>
      <add name="System.Net"
        type="System.Diagnostics.TextWriterTraceListener"
        initializeData="network.log"
      />
    </sharedListeners>
    <trace autoflush="true"/>
  </system.diagnostics>

Resulting log

System.Net Information: 0 : [15028] Current OS installation type is 'Client'.
System.Net Verbose: 0 : [15028] HttpWebRequest#40383808::HttpWebRequest(http://oak:8080/bigresource#-236952546)
System.Net Information: 0 : [15028] RAS supported: True
System.Net Verbose: 0 : [15028] Exiting HttpWebRequest#40383808::HttpWebRequest() 
System.Net Verbose: 0 : [15028] HttpWebRequest#40383808::HttpWebRequest(uri: 'http://oak:8080/bigresource', connectionGroupName: '17480744')
System.Net Verbose: 0 : [15028] Exiting HttpWebRequest#40383808::HttpWebRequest() 
System.Net Verbose: 0 : [25748] HttpWebRequest#40383808::BeginGetResponse()
System.Net Verbose: 0 : [25748] ServicePoint#45653674::ServicePoint(127.0.0.1:8888)
System.Net Information: 0 : [25748] Associating HttpWebRequest#40383808 with ServicePoint#45653674
System.Net Information: 0 : [25748] Associating Connection#41149443 with HttpWebRequest#40383808
System.Net Verbose: 0 : [25748] Exiting HttpWebRequest#40383808::BeginGetResponse()     -> ContextAwareResult#39785641
System.Net Information: 0 : [3264] Connection#41149443 - Created connection from 127.0.0.1:10411 to 127.0.0.1:8888.
System.Net Information: 0 : [3264] Associating HttpWebRequest#40383808 with ConnectStream#39086322
System.Net Information: 0 : [3264] HttpWebRequest#40383808 - Request: GET http://oak:8080/bigresource HTTP/1.1

System.Net Information: 0 : [3264] ConnectStream#39086322 - Sending headers
{
Host: oak:8080
Proxy-Connection: Keep-Alive
}.
System.Net Information: 0 : [21384] Connection#41149443 - Received status line: Version=1.1, StatusCode=200, StatusDescription=OK.
System.Net Information: 0 : [21384] Connection#41149443 - Received headers
{
Content-Length: 48890
Content-Type: text/plain; charset=utf-8
Date: Thu, 09 Jan 2014 16:41:59 GMT
Server: Microsoft-HTTPAPI/2.0
}.
System.Net Information: 0 : [21384] ConnectStream#56140151::ConnectStream(Buffered 48890 bytes.)
System.Net Information: 0 : [21384] Associating HttpWebRequest#40383808 with ConnectStream#56140151
System.Net Information: 0 : [21384] Associating HttpWebRequest#40383808 with HttpWebResponse#1997173
System.Net Verbose: 0 : [21384] HttpWebRequest#40383808::EndGetResponse()
System.Net Verbose: 0 : [21384] Exiting HttpWebRequest#40383808::EndGetResponse()   -> HttpWebResponse#1997173
System.Net Verbose: 0 : [21384] HttpWebResponse#1997173::GetResponseStream()
System.Net Information: 0 : [21384] ContentLength=48890
System.Net Verbose: 0 : [21384] Exiting HttpWebResponse#1997173::GetResponseStream()    -> ConnectStream#56140151
System.Net Verbose: 0 : [15028] ConnectStream#56140151::Read()
System.Net Verbose: 0 : [15028] Data from ConnectStream#56140151::Read
System.Net Verbose: 0 : [15028] 00000000 : 30 2C 31 2C 32 2C 33 2C-34 2C 35 2C 36 2C 37 2C : 0,1,2,3,4,5,6,7,
System.Net Verbose: 0 : [15028] 00000010 : 38 2C 39 2C 31 30 2C 31-31 2C 31 32 2C 31 33 2C : 8,9,10,11,12,13,
System.Net Verbose: 0 : [15028] 00000020 : 31 34 2C 31 35 2C 31 36-2C 31 37 2C 31 38 2C 31 : 14,15,16,17,18,1
System.Net Verbose: 0 : [15028] 00000030 : 39 2C 32 30 2C 32 31 2C-32 32 2C 32 33 2C 32 34 : 9,20,21,22,23,24
System.Net Verbose: 0 : [15028] 00000040 : 2C 32 35 2C 32 36 2C 32-37 2C 32 38 2C 32 39 2C : ,25,26,27,28,29,
System.Net Verbose: 0 : [15028] 00000050 : 33 30 2C 33 31 2C 33 32-2C 33 33 2C 33 34 2C 33 : 30,31,32,33,34,3
System.Net Verbose: 0 : [15028] 00000060 : 35 2C 33 36 2C 33 37 2C-33 38 2C 33 39 2C 34 30 : 5,36,37,38,39,40
System.Net Verbose: 0 : [15028] 00000070 : 2C 34 31 2C 34 32 2C 34-33 2C 34 34 2C 34 35 2C : ,41,42,43,44,45,
System.Net Verbose: 0 : [15028] 00000080 : 34 36 2C 34 37 2C 34 38-2C 34 39 2C 35 30 2C 35 : 46,47,48,49,50,5
System.Net Verbose: 0 : [15028] 00000090 : 31 2C 35 32 2C 35 33 2C-35 34 2C 35 35 2C 35 36 : 1,52,53,54,55,56
System.Net Verbose: 0 : [15028] 000000A0 : 2C 35 37 2C 35 38 2C 35-39 2C 36 30 2C 36 31 2C : ,57,58,59,60,61,
System.Net Verbose: 0 : [15028] 000000B0 : 36 32 2C 36 33 2C 36 34-2C 36 35 2C 36 36 2C 36 : 62,63,64,65,66,6
System.Net Verbose: 0 : [15028] 000000C0 : 37 2C 36 38 2C 36 39 2C-37 30 2C 37 31 2C 37 32 : 7,68,69,70,71,72
System.Net Verbose: 0 : [15028] 000000D0 : 2C 37 33 2C 37 34 2C 37-35 2C 37 36 2C 37 37 2C : ,73,74,75,76,77,
System.Net Verbose: 0 : [15028] 000000E0 : 37 38 2C 37 39 2C 38 30-2C 38 31 2C 38 32 2C 38 : 78,79,80,81,82,8
System.Net Verbose: 0 : [15028] 000000F0 : 33 2C 38 34 2C 38 35 2C-38 36 2C 38 37 2C 38 38 : 3,84,85,86,87,88
System.Net Verbose: 0 : [15028] 00000100 : 2C 38 39 2C 39 30 2C 39-31 2C 39 32 2C 39 33 2C : ,89,90,91,92,93,
System.Net Verbose: 0 : [15028] 00000110 : 39 34 2C 39 35 2C 39 36-2C 39 37 2C 39 38 2C 39 : 94,95,96,97,98,9
System.Net Verbose: 0 : [15028] 00000120 : 39 2C 31 30 30 2C 31 30-31 2C 31 30 32 2C 31 30 : 9,100,101,102,10
System.Net Verbose: 0 : [15028] 00000130 : 33 2C 31 30 34 2C 31 30-35 2C 31 30 36 2C 31 30 : 3,104,105,106,10
System.Net Verbose: 0 : [15028] 00000140 : 37 2C 31 30 38 2C 31 30-39 2C 31 31 30 2C 31 31 : 7,108,109,110,11
System.Net Verbose: 0 : [15028] 00000150 : 31 2C 31 31 32 2C 31 31-33 2C 31 31 34 2C 31 31 : 1,112,113,114,11
System.Net Verbose: 0 : [15028] 00000160 : 35 2C 31 31 36 2C 31 31-37 2C 31 31 38 2C 31 31 : 5,116,117,118,11
System.Net Verbose: 0 : [15028] 00000170 : 39 2C 31 32 30 2C 31 32-31 2C 31 32 32 2C 31 32 : 9,120,121,122,12
System.Net Verbose: 0 : [15028] 00000180 : 33 2C 31 32 34 2C 31 32-35 2C 31 32 36 2C 31 32 : 3,124,125,126,12
System.Net Verbose: 0 : [15028] 00000190 : 37 2C 31 32 38 2C 31 32-39 2C 31 33 30 2C 31 33 : 7,128,129,130,13
System.Net Verbose: 0 : [15028] 000001A0 : 31 2C 31 33 32 2C 31 33-33 2C 31 33 34 2C 31 33 : 1,132,133,134,13
System.Net Verbose: 0 : [15028] 000001B0 : 35 2C 31 33 36 2C 31 33-37 2C 31 33 38 2C 31 33 : 5,136,137,138,13
System.Net Verbose: 0 : [15028] 000001C0 : 39 2C 31 34 30 2C 31 34-31 2C 31 34 32 2C 31 34 : 9,140,141,142,14
System.Net Verbose: 0 : [15028] 000001D0 : 33 2C 31 34 34 2C 31 34-35 2C 31 34 36 2C 31 34 : 3,144,145,146,14
System.Net Verbose: 0 : [15028] 000001E0 : 37 2C 31 34 38 2C 31 34-39 2C 31 35 30 2C 31 35 : 7,148,149,150,15
System.Net Verbose: 0 : [15028] 000001F0 : 31 2C 31 35 32 2C 31 35-33 2C 31 35 34 2C 31 35 : 1,152,153,154,15
System.Net Verbose: 0 : [15028] 00000200 : 35 2C 31 35 36 2C 31 35-37 2C 31 35 38 2C 31 35 : 5,156,157,158,15
System.Net Verbose: 0 : [15028] 00000210 : 39 2C 31 36 30 2C 31 36-31 2C 31 36 32 2C 31 36 : 9,160,161,162,16
System.Net Verbose: 0 : [15028] 00000220 : 33 2C 31 36 34 2C 31 36-35 2C 31 36 36 2C 31 36 : 3,164,165,166,16
System.Net Verbose: 0 : [15028] 00000230 : 37 2C 31 36 38 2C 31 36-39 2C 31 37 30 2C 31 37 : 7,168,169,170,17
System.Net Verbose: 0 : [15028] 00000240 : 31 2C 31 37 32 2C 31 37-33 2C 31 37 34 2C 31 37 : 1,172,173,174,17
System.Net Verbose: 0 : [15028] 00000250 : 35 2C 31 37 36 2C 31 37-37 2C 31 37 38 2C 31 37 : 5,176,177,178,17
System.Net Verbose: 0 : [15028] 00000260 : 39 2C 31 38 30 2C 31 38-31 2C 31 38 32 2C 31 38 : 9,180,181,182,18
System.Net Verbose: 0 : [15028] 00000270 : 33 2C 31 38 34 2C 31 38-35 2C 31 38 36 2C 31 38 : 3,184,185,186,18
System.Net Verbose: 0 : [15028] 00000280 : 37 2C 31 38 38 2C 31 38-39 2C 31 39 30 2C 31 39 : 7,188,189,190,19
System.Net Verbose: 0 : [15028] 00000290 : 31 2C 31 39 32 2C 31 39-33 2C 31 39 34 2C 31 39 : 1,192,193,194,19
System.Net Verbose: 0 : [15028] 000002A0 : 35 2C 31 39 36 2C 31 39-37 2C 31 39 38 2C 31 39 : 5,196,197,198,19
System.Net Verbose: 0 : [15028] 000002B0 : 39 2C 32 30 30 2C 32 30-31 2C 32 30 32 2C 32 30 : 9,200,201,202,20
System.Net Verbose: 0 : [15028] 000002C0 : 33 2C 32 30 34 2C 32 30-35 2C 32 30 36 2C 32 30 : 3,204,205,206,20
System.Net Verbose: 0 : [15028] 000002D0 : 37 2C 32 30 38 2C 32 30-39 2C 32 31 30 2C 32 31 : 7,208,209,210,21
System.Net Verbose: 0 : [15028] 000002E0 : 31 2C 32 31 32 2C 32 31-33 2C 32 31 34 2C 32 31 : 1,212,213,214,21
System.Net Verbose: 0 : [15028] 000002F0 : 35 2C 32 31 36 2C 32 31-37 2C 32 31 38 2C 32 31 : 5,216,217,218,21
System.Net Verbose: 0 : [15028] 00000300 : 39 2C 32 32 30 2C 32 32-31 2C 32 32 32 2C 32 32 : 9,220,221,222,22
System.Net Verbose: 0 : [15028] 00000310 : 33 2C 32 32 34 2C 32 32-35 2C 32 32 36 2C 32 32 : 3,224,225,226,22
System.Net Verbose: 0 : [15028] 00000320 : 37 2C 32 32 38 2C 32 32-39 2C 32 33 30 2C 32 33 : 7,228,229,230,23
System.Net Verbose: 0 : [15028] 00000330 : 31 2C 32 33 32 2C 32 33-33 2C 32 33 34 2C 32 33 : 1,232,233,234,23
System.Net Verbose: 0 : [15028] 00000340 : 35 2C 32 33 36 2C 32 33-37 2C 32 33 38 2C 32 33 : 5,236,237,238,23
System.Net Verbose: 0 : [15028] 00000350 : 39 2C 32 34 30 2C 32 34-31 2C 32 34 32 2C 32 34 : 9,240,241,242,24
System.Net Verbose: 0 : [15028] 00000360 : 33 2C 32 34 34 2C 32 34-35 2C 32 34 36 2C 32 34 : 3,244,245,246,24
System.Net Verbose: 0 : [15028] 00000370 : 37 2C 32 34 38 2C 32 34-39 2C 32 35 30 2C 32 35 : 7,248,249,250,25
System.Net Verbose: 0 : [15028] 00000380 : 31 2C 32 35 32 2C 32 35-33 2C 32 35 34 2C 32 35 : 1,252,253,254,25
System.Net Verbose: 0 : [15028] 00000390 : 35 2C 32 35 36 2C 32 35-37 2C 32 35 38 2C 32 35 : 5,256,257,258,25
System.Net Verbose: 0 : [15028] 000003A0 : 39 2C 32 36 30 2C 32 36-31 2C 32 36 32 2C 32 36 : 9,260,261,262,26
System.Net Verbose: 0 : [15028] 000003B0 : 33 2C 32 36 34 2C 32 36-35 2C 32 36 36 2C 32 36 : 3,264,265,266,26
System.Net Verbose: 0 : [15028] 000003C0 : 37 2C 32 36 38 2C 32 36-39 2C 32 37 30 2C 32 37 : 7,268,269,270,27
System.Net Verbose: 0 : [15028] 000003D0 : 31 2C 32 37 32 2C 32 37-33 2C 32 37 34 2C 32 37 : 1,272,273,274,27
System.Net Verbose: 0 : [15028] 000003E0 : 35 2C 32 37 36 2C 32 37-                        : 5,276,27
System.Net Verbose: 0 : [15028] Exiting ConnectStream#56140151::Read()  -> Int32#1000