HttpWebRequest partially fails to get data

Question

I have this code, calling this sample API call of the game Guild Wars 2

        HttpWebRequest request = HttpWebRequest.Create("https://api.guildwars2.com/v1/events.json?world_id=1001") as HttpWebRequest;
        HttpWebResponse response = request.GetResponse() as HttpWebResponse;

        byte[] bytes = new byte[response.ContentLength];
        Stream stream = response.GetResponseStream();
        stream.Read(bytes, 0, bytes.Length);
        string result = Encoding.Default.GetString(bytes);

It is a sample call to the official Guild Wars 2 API and is supposed to return a list of events and their status in JSON format.

If you call the address in a browser and paste the data into a text editor, it gives a correct JSON string, of a little more than 300 kb.

However, when called by this code and, looking at the resulting byte array, the first 3800-3900 bytes are correctly filled (the number varies slightly from call to call), but the rest is all zero.

The response.ContentLength indicates the correct length of the stream, a little more than 300k, and testing with stream.ReadByte() tells me the stream does deliver those 300k, it's just zeroes after that 3850 byte mark.

Is there an error in my code, or what could cause this problem? Could I call the API in another way to get the correct, full response?

possible duplicate of [What is the best way to read GetResponseStream()?](http://stackoverflow.com/questions/137285/what-is-the-best-way-to-read-getresponsestream) — CodeCaster, Oct 23 '13 at 12:11

C.Evenhuis · Accepted Answer · 2013-10-23T12:10:54.037

4

The Read operation on a stream returns the number of bytes actually read. You should keep calling stream.Read() until it returns 0. The data simply has not arrived yet.

The reason Read won't block is that some applications may already begin processing the partial data.

You could use the following code:

HttpWebRequest request = HttpWebRequest.Create("https://api.guildwars2.com/v1/events.json?world_id=1001") as HttpWebRequest;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;

byte[] bytes;
using (Stream stream = response.GetResponseStream())
using (MemoryStream buffer = new MemoryStream((int)response.ContentLength))
{
    byte[] chunk = new byte[4096];
    int bytesRead;
    while ((bytesRead = stream.Read(chunk, 0, chunk.Length)) > 0)
    {
        buffer.Write(chunk, 0, bytesRead);
    }
    bytes = buffer.ToArray();
}

edited Oct 23 '13 at 12:10

answered Oct 23 '13 at 12:05

C.Evenhuis

25,996
2
58
72

Thanks, that worked. What's the background on this? Is that a limitation on the API side, like speed throttling, or chunk size? – Hackworth Oct 23 '13 at 12:17
1

The reason for `Read()` to return early is that some applications (for streamed video, for example) can then start processing data before the request has completed. The chunk size of 4096 in my example might just as well be any value. That value should be smaller than the largest chunk of data you'll ever receive at once, but if it's too small the code will waste processing power. – C.Evenhuis Oct 23 '13 at 12:24

HttpWebRequest partially fails to get data

1 Answers1