WebException.Response.GetResponseStream() limited to 65536 characters

Question

I am trying to retrieve HTML code from a webpage using HttpWebRequest and HttpWebResponse.

response = (HttpWebResponse)request.GetResponse();
...
Stream stream = response.GetResponseStream();

The response object has a ContentLength value of 106142. When I look at the stream object, it has a length of 65536. When reading the stream with a StreamReader using ReadToEnd(), only the first 65536 characters are returned.

How can I get the whole code?

Edit:

Using the following code segment:

catch (WebException ex)
{
    errorMessage = errorMessage + ex.Message;
    if (ex.Response != null) {
        if (ex.Response.ContentLength > 0) 
        {
            using (Stream stream = ex.Response.GetResponseStream())
            {
                using (StreamReader reader = new StreamReader(stream))
                {
                    string pageOutput = reader.ReadToEnd().Trim();

ex.Response.ContentLength = 106142

ex.Response.GetResponseStream().Length = 65536

stream.Length = 65536

pageOutput.Length = 65534 (because of the trim)

And yes, the code is actually truncated.

Have you looked at the result string? Does it look like there is something missing? Depending on the encoding used, ContentLength (in bytes) might not match the # of characters returned from ReadToEnd() (i.e. if it's using two bytes per char, etc.) — blech, Feb 01 '11 at 17:22
thanks for posting the actual problem. So, the problem is that the response body from a WebException is truncated. — John Saunders, Feb 01 '11 at 19:37
For future visitors: To actually resolve this issue rather than hope it doesn't happen to you; please see http://stackoverflow.com/questions/4918107/system-net-httpwebresponse-getresponsestream-returns-truncated-body-in-webexce — Mark Henderson, Oct 03 '13 at 04:45

score 7 · Answer 1 · edited May 23 '17 at 12:02

7

You can find an answer in this topic in System.Net.HttpWebResponse.GetResponseStream() returns truncated body in WebException

You have to manage the HttpWebRequest object and change DefaultMaximumErrorResponseLength property. For example :

HttpWebRequest.DefaultMaximumErrorResponseLength = 1048576;

edited May 23 '17 at 12:02

Community

1
1

answered May 15 '14 at 08:24

Alexein1

109
1
6

score 1 · Answer 2 · answered Feb 01 '11 at 17:16

1

ReadToEnd does specifically just that, it reads to the end of the stream. I would check to make sure that you were actually being sent the entire expected response.

answered Feb 01 '11 at 17:16

khr055

28,690
16
36
48

score -1 · Accepted Answer · answered Feb 01 '11 at 19:36

There seems to be a problem when calling the GetResponseStream() method on the HttpWebResponse returned by the exception. Everything works as expected when there is no exception.

I wanted to get the HTML code from the error returned by the server.

I guess I'll have to hope the error doesn't exceed 65536 characters...

WebException.Response.GetResponseStream() limited to 65536 characters

3 Answers3

Linked