HttpWebRequest gets slower when adding an Interval

Question

Testing different possibilities to download the source of a webpage I got the following results (Average time in ms to google.com, 9gag.com):

Plain HttpWebRequest: 169, 360
Gzip HttpWebRequest: 143, 260
WebClient GetStream : 132, 295
WebClient DownloadString: 143, 389

So for my 9gag client I decided to take the gzip HttpWebRequest. The problem is, after implementing in my actual program, the request takes more than twice the time.
The Problem also occurs when just adding a Thread.Sleep between two requests.

EDIT:
Just improved the code a bit, still the same problem: When running in a loop the requests takes longer when I add an Delay between to requests

for(int i = 0; i < 100; i++)
{
    getWebsite("http://9gag.com/");
}

Takes about 250ms per request.

for(int i = 0; i < 100; i++)
{
    getWebsite("http://9gag.com/");
    Thread.Sleep(1000);
}

Takes about 610ms per request.

    private string getWebsite(string Url)
    {
        Stopwatch stopwatch = Stopwatch.StartNew();

        HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
        http.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        string html = string.Empty;

        using (HttpWebResponse webResponse = (HttpWebResponse)http.GetResponse())
        using (Stream responseStream = webResponse.GetResponseStream())
        using (StreamReader reader = new StreamReader(responseStream))
        {

            html = reader.ReadToEnd();
        }

        Debug.WriteLine(stopwatch.ElapsedMilliseconds);
        return html;
    }

Any ideas to fix this problem?

Just in case can you remove the call to Debug.WriteLine inside your looped function? Time the 100 iterations as a block and divide total time by 100. Should be roughly equivalent. Just a sanity check to make sure that the writeline isn't slowing you down at all. I'm pretty sure it's supposed to be fast, but doesn't hurt to confirm. — Kenneth Ito, Jul 07 '12 at 23:03

score 5 · Answer 1 · answered Jul 07 '12 at 23:07

Maybe give this a try, although it might only help your case of a single request and actually make things worse when doing a multithreaded version.

ServicePointManager.UseNagleAlgorithm = false;

Here's a quote from MSDN docs for the HttpWebRequest Class

Another option that can have an impact on performance is the use of the UseNagleAlgorithm property. When this property is set to true, TCP/IP will try to use the TCP Nagle algorithm for HTTP connections. The Nagle algorithm aggregates data when sending TCP packets. It accumulates sequences of small messages into larger TCP packets before the data is sent over the network. Using the Nagle algorithm can optimize the use of network resources, although in some situations performance can also be degraded. Generally for constant high-volume throughput, a performance improvement is realized using the Nagle algorithm. But for smaller throughput applications, degradation in performance may be seen.

An application doesn't normally need to change the default value for the UseNagleAlgorithm property which is set to true. However, if an application is using low-latency connections, it may help to set this property to false.

You my friend, are fantastic. Thanks. – Nathan Smith Sep 23 '13 at 12:47 — Nathan Smith, Sep 23 '13 at 12:47

score 1 · Answer 2 · answered Jul 06 '12 at 12:35

I think you might be leaking resources as you aren't disposing of all of your IDisposable object with each method call.

Give this version and try and see if it gives you a more consistent execution time.

public string getWebsite( string Url )
  {
     Stopwatch stopwatch = Stopwatch.StartNew();

     HttpWebRequest http = (HttpWebRequest) WebRequest.Create( Url );
     http.Headers.Add( HttpRequestHeader.AcceptEncoding, "gzip,deflate" );

     string html = string.Empty;
     using ( HttpWebResponse webResponse = (HttpWebResponse) http.GetResponse() )
     {
        using ( Stream responseStream = webResponse.GetResponseStream() )
        {
           Stream decompressedStream = null;

           if ( webResponse.ContentEncoding.ToLower().Contains( "gzip" ) )
              decompressedStream = new GZipStream( responseStream, CompressionMode.Decompress );
           else if ( webResponse.ContentEncoding.ToLower().Contains( "deflate" ) )
              decompressedStream = new DeflateStream( responseStream, CompressionMode.Decompress );

           if ( decompressedStream != null )
           {
              using ( StreamReader reader = new StreamReader( decompressedStream, Encoding.Default ) )
              {
                 html = reader.ReadToEnd();
              }

              decompressedStream.Dispose();
           }
        }
     }

     Debug.WriteLine( stopwatch.ElapsedMilliseconds );

     return html;
  }

Thanks for the fast reply, I didn't notice that. Tried your version but no improvement. When I call the method 100 times in a row each request takes now almost exactly 250ms. When I add a delay of a second between two requests, one request takes 610ms. — M4a1x, Jul 06 '12 at 13:28

HttpWebRequest gets slower when adding an Interval

2 Answers2

Linked