Speed of HttpWebRequest/HttpWebResponse

Question

Is there any quicker alternative to the code below to get a http response into a string?

string req = "http://someaddress.com";
Stopwatch timer = new Stopwatch();
timer.Start();
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
      using (Stream dataStream = response.GetResponseStream())
      {
            StreamReader reader = new StreamReader(dataStream);
            reader.ReadToEnd();
      }
}
timer.Stop();
Console.WriteLine(timer.Elapsed);

The response is pretty big - around 2MB and is in XML format. Affter this codes completes, the timer is equal to ~50 seconds. When I paste the same url into the browser window it takes about 35 seconds for it to display the xml document.

What and how are you measuring? Does it take 50 seconds to write the XML with `Console.WriteLine()` or what do you do with it? And does the browser use 35 seconds to render the XML as plain text or does it format the XML in any way? I think you're comparing apples to oranges. — Asbjørn Ulsberg, Sep 29 '11 at 10:23
As Jon Skeet writes, the time might be spent in decoding the bytes into a string in the `reader.ReadToEnd()` method. You have also still not provided any information on what it is you're measuring in your web browser. — Asbjørn Ulsberg, Sep 29 '11 at 11:14
What about "When I paste the same url into the browser window it takes about 35 seconds for it to display the xml document."? — stiopa, Sep 29 '11 at 12:13

Jon Skeet · Accepted Answer · 2011-09-29T10:32:28.800

3

(You should have a using statement for the response, by the way... and I agree with asbjornu's comment. You should update your question with more details.)

You should use something like Wireshark to look at what the requests and responses look like in each case. For example, is the browser specifying that it supports compressed responses, and WebRequest not? If it's over a slow connection, that could well be the important part.

Another thing to test is whether the string decoding is taking significant time in the .NET code... if you simply read the data from the stream into a byte array (possibly just throwing it away as you read it) is that significantly faster? For example:

using (var response = request.GetResponse())
{
    using (var stream = response.GetResponseStream())
    {
        // Just read the data and throw it away
        byte[] buffer = new byte[16 * 1024];
        int bytesRead;
        while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
        {
            // Ignore the actual data
        }
    }
}

edited Sep 29 '11 at 10:32

answered Sep 29 '11 at 10:24

Jon Skeet

1,421,763
867
9,128
9,194

Browser request: GET somerequest HTTP/1.1 Host: somehost User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.22) Gecko/20110902 Firefox/3.6.22 (.NET CLR 3.5.30729) Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive – stiopa Sep 29 '11 at 12:18
@stiopa: Right: not the `Accept-Encoding: gzip,default`. Now what does WebRequest do? – Jon Skeet Sep 29 '11 at 12:33
Webrequest: GET some address HTTP/1.1 Host: somehost ..Connection: Keep-Alive.... Is there any way to get a compressed response with Webrequest then? – stiopa Sep 29 '11 at 12:41
1

@stiopa: See http://stackoverflow.com/questions/678547/does-nets-httpwebresponse-uncompress-automatically-gziped-and-deflated-response – Jon Skeet Sep 29 '11 at 12:48
And it's down to 33 seconds. Thanks for your help Jon. – stiopa Sep 29 '11 at 12:53

Speed of HttpWebRequest/HttpWebResponse

1 Answers1