How to make HttpClient ignore Content-Length header

Question

I am using HttpClient to communicate with a server which I don't have access to. Sometimes the JSON response from the server is truncated.

The problem occurs when the Content-Length header is smaller than what it should be (8192 vs. 8329). It seems like a bug on the server which gives a smaller Content-Length header than the actual size of the response body. If I use Google Chrome instead of HttpClient, the response is always complete.

Therefore, I want to make HttpClient to ignore the wrong Content-Length header and read to the end of the response. Is it possible to do that? Any other solution is well appreciated. Thank you!

This is the code of my HttpClient:

var client = new HttpClient();
client.BaseAddress = new Uri(c_serverBaseAddress);

HttpResponseMessage response = null;
try
{
      response = await client.GetAsync(c_serverEventApiAddress + "?location=" + locationName);
}
catch (Exception e)
{
    // Do something
}
var json = response.Content.ReadAsStringAsync().Result;

var obj = JsonConvert.DeserializeObject<JObject>(json); // The EXCEPTION occurs HERE!!! Because the json is truncated!

EDIT 1:

If I use HttpWebRequest, it can read to the end of the JSON response completely without any truncation. However, I would like to use HttpClient since it has better async/await.

This is the code using HttpWebRequest:

var url = c_serverBaseAddress + c_serverEventApiAddress + "?location=" + "Saskatchewan";

var request = (HttpWebRequest)WebRequest.Create(url); 
request.ProtocolVersion = HttpVersion.Version10;
request.Method = "GET";
request.ContentType = "application/x-www-form-urlencoded";

var response = (HttpWebResponse)request.GetResponse();

StringBuilder stringBuilder = new StringBuilder(); 
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
      string line;
      while ((line = reader.ReadLine()) != null)
      {
            stringBuilder.Append(line);
      }
}
var json = stringBuilder.ToString();  // COMPLETE json response everytime!!!

So the same call with Chrome shows a valid JSON response, but with HttpClient you get an incomplete JSON object in that `json` variable from your example? — vtortola, Nov 02 '15 at 13:16
Correct. Sometimes HttpClient returns an incomplete JSON response, sometimes it returns a complete one. The incomplete JSON always comes together with a smaller Content-Length header. — nxh, Nov 02 '15 at 15:48
And the server is returning the whole JSON response with a wrong Content-Length, or the response is actually truncated? — vtortola, Nov 02 '15 at 16:30
The server returns the whole JSON. Chrome retrieves it without any problem, and if I use HttpWebRequest it can read to the end of the complete JSON. I am updating the question to include the code using HttpWebRequest. — nxh, Nov 02 '15 at 16:50
I end up using HttpWebRequest as a workaround. Using TaskFactory.FromAsync, making asynchronous calls with HttpWebRequest is quite easy. Reference: http://stackoverflow.com/questions/202481/how-to-use-httpwebrequest-net-asynchronously/23004036#23004036 — nxh, Nov 03 '15 at 09:13
Note: it is a bit unrelated to the question but I found some weird behaviors of the server which may help someone coping with similar issues. Testing with curl, the server actually doesn't return Content-Length header. However response.Content.ContentLength always gives a number. HttpWebRequest always returns truncated json if I use Http 1.1, full json if use 1.0. Testing with curl forcing Http 1.1, the response is returned completely in 2 chunks: first 2000 bytes and second 89 bytes. The truncated json is exactly the first chunk. — nxh, Nov 04 '15 at 09:14
Therefore, the truncation happens maybe because HttpWebRequest (and HttpClient) ignores all other chunks except the first one. That's the reason why using Http 1.0 solve the problem since Http 1.0 doesn't have chunked-encoding. Btw, my code is actually running on Mono on a Mac since I am programming with Xamarin (iOS and Android) — nxh, Nov 04 '15 at 09:17
I have just found out that HttpWebRequest doens't support Chunked-Encoding and HttpClient does. http://stackoverflow.com/questions/14932183/transfer-encoding-chunked-in-windows-phone. The strange thing in my case is HttpClient receives only the first chunk. — nxh, Nov 04 '15 at 09:40
Another weird thing is: The first chunk length is 2000 bytes according to curl. However, if I use HttpClient, I will receive only the same chunk but response.Content.Headers.ContentLength is 8192. — nxh, Nov 04 '15 at 10:03
Have you tried setting this client.DefaultRequestHeaders.TransferEncodingChunked explicitly? Also you should be able to remove all headers from client.DefaultRequestHeaders collection and specify the ones you need? I would also make that sync call with .result an async await just in case its causing odd behaviour. — Netferret, Feb 22 '21 at 17:23

score 1 · Answer 1 · answered Nov 23 '21 at 02:12

You could try specifying the buffer size of the response content

var client = new HttpClient();
client.MaxResponseContentBufferSize = <your_buffer_size>;

Where the property MaxResponseContentBufferSize means:

Gets or sets the maximum number of bytes to buffer when reading the response content.

How to make HttpClient ignore Content-Length header

1 Answers1

Linked