HttpWebRequest is extremely slow!

Question

I am using an open source library to connect to my webserver. I was concerned that the webserver was going extremely slow and then I tried doing a simple test in Ruby and I got these results

Ruby program: 2.11seconds for 10 HTTP GETs

Ruby program: 18.13seconds for 100 HTTP GETs

C# library: 20.81seconds for 10 HTTP GETs

C# library: 36847.46seconds for 100 HTTP GETs

I have profiled and found the problem to be this function:

private HttpWebResponse GetRawResponse(HttpWebRequest request) {
  HttpWebResponse raw = null;
  try {
    raw = (HttpWebResponse)request.GetResponse(); //This line!
  }
  catch (WebException ex) {
    if (ex.Response is HttpWebResponse) {
      raw = ex.Response as HttpWebResponse;
    }
  }
  return raw;
}

The marked line is takes over 1 second to complete by itself while the ruby program making 1 request takes .3 seconds. I am also doing all of these tests on 127.0.0.1, so network bandwidth is not an issue.

What could be causing this huge slow down?

UPDATE

Check out the changed benchmark results. I actually tested with 10 GETs and not 100, I updated the results.

Out of curiosity, are you using the same HttpWebRequest object for all GETs? I think the slowdown is just the initial handshake. Try reusing the same HttpWebRequest object. — BFree, Mar 25 '10 at 22:10
When you profile, do you exclude the first call? I have sometimes noticed the FireWall (or something) can slow down the 1st call by seconds. (Alt: measure more than 100 requests, does the ratio stay the same?) — H H, Mar 25 '10 at 22:13
I believe that it is creating a new request object on each call.. What kind of "handshake" must be done for HTTP though? its a fairly simplistic protocol — Earlz, Mar 25 '10 at 22:14
Also, my ruby test program creates a new HTTP object each time as well, so apparently HTTP doesn't have to be slow to create new objects each time. — Earlz, Mar 25 '10 at 22:16
@Henk, I just tried doing 500 requests and gave up after 5 minutes... — Earlz, Mar 25 '10 at 22:24
Can you show the code that is creating and initializing the webrequest? Are you using any authentication? SSL? — feroze, Mar 26 '10 at 01:21
No authentication or SSL.. I figured out there was something wrong with my test code though.. just reran the ruby program and now I can do 100 requests in 2 seconds... idk.. I think this problem is actually too localized.. — Earlz, Mar 26 '10 at 02:39
1 http://haacked.com/archive/2004/05/15/http-web-request-expect-100-continue.aspx 2 [http://blogs.msdn.com/b/joncole/archive/2005/09/08/462659.aspx](http://blogs.msdn.com/b/joncole/archive/2005/09/08/462659.aspx) — , Jul 12 '11 at 11:51
I have also been experiencing this with request.GetResponse() taking over 20s for a simple GET/POST. I found disabling IP V6 changed the result to < 1s. Not sure why but interested if others find the same. — chrisb, Oct 10 '15 at 17:35
@chrisb how do you do that, disable ip v6 on the HttpWebResponse. I found this https://stackoverflow.com/questions/37384945/how-to-force-ipv6-or-ipv4-for-httpwebrequest-or-webrequest-c-sharp. Is this something you have in mind? — Sotiris Zegiannis, Feb 17 '19 at 14:52

score 177 · Accepted Answer · edited Sep 03 '13 at 10:02

177

What I have found to be the main culprit with slow web requests is the proxy property. If you set this property to null before you call the GetResponse method the query will skip the proxy autodetect step:

request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
}

The proxy autodetect was taking up to 7 seconds to query before returning the response. It is a little annoying that this property is set on by default for the HttpWebRequest object.

edited Sep 03 '13 at 10:02

Dunc

18,404
6
86
103

answered Aug 30 '10 at 19:08

James Roland

8,183
4
30
16

2

James, you rock. I'd have found this eventually via Wireshark but literally seconds after reading this post our WebDAV performance was over one hundred faster. I suspect our Sharpoint server has the same issue – Paul Lockwood Feb 14 '12 at 21:15
1

I'm having the same issue (response taking more or less 1 second) but setting WebRequest.DefaultWebProxy = null doesn't solve my issue. Any other idea? – Cristiano Ghersi Mar 10 '13 at 20:03
Sorry for bumping the thread up, but what happens if the address you try to access is only available through the default proxy ? – Florian F. Mar 20 '13 at 08:59
You might also consider not setting `request.Proxy` to `null` (in case the user sits behind a proxy) and let the first request more time to resolve the proxy, and reuse the same proxy instance for every subsequent request... – Pierre Arnaud Aug 09 '13 at 05:32
3

I suspect the key here is rather to wrap the `GetResponse` call into a `using` directive as there's a limit to the number of connections and you have to wait for them to time out if you don't close them properly. – Nicolas78 Aug 12 '13 at 08:38
In my case wrapping it in a 'using' directive was the key. – craigmoliver Aug 16 '13 at 20:42
Is there a way to detect if a proxy was found? -- I would like to conditionally only set this to null after the first request (and only if the proxy wasn't found) -- on the other side of the coin -- if a proxy WAS found -- can I cache that, so that it doesn't spend 4 seconds refinding it every call? – BrainSlugs83 Jan 12 '14 at 04:21
This doesn't work much for me. Manu's solution is more suitable for me. My scenario is like this: http://stackoverflow.com/questions/24154101/how-to-inprove-the-performance-of-i-o-bound-multi-threading – smwikipedia Jun 12 '14 at 02:37
1

This worked for me, thanks a lot. Just using HTTPWebRequest's GetResponse method was taking 5+ minutes to respond. It tries to get the system proxy by default and times out. Manually calling GetSystemWebProxy also had the same issue on my machine. But setting proxy null made it fast again. – bobasaurus Sep 06 '17 at 23:47
HttpClient demostrates the same behavior, it delays every first request, even if you use it as singleton. The key is networ autodiscovery request. You should configure proxy via HttpClientHandler and set UseProxy to false and Proxy to null. – Jonik Aug 08 '19 at 07:09

score 23 · Answer 2 · answered Mar 25 '10 at 22:16

23

It may have to do with the fact that you are opening several connections at once. By default the Maximum amount of open HTTP connections is set to two. Try adding this to your .config file and see if it helps:

<system.net>
  .......
  <connectionManagement>
    <add address="*" maxconnection="20"/>
  </connectionManagement>
</system.net>

answered Mar 25 '10 at 22:16

Manu

28,753
28
75
83

This is the one works for me. My scenario is like this: http://stackoverflow.com/questions/24154101/how-to-inprove-the-performance-of-i-o-bound-multi-threading – smwikipedia Jun 12 '14 at 02:37

score 11 · Answer 3 · answered Jan 15 '13 at 16:26

I was having a similar issue with a VB.Net MVC project.
Locally on my pc (Windows 7) it was taking under 1 second to hit the page requests, but on the server (Windows Server 2008 R2) it was taking 20+ seconds for each page request.

I tried a combination of setting the proxy to null

  System.Net.WebRequest.DefaultWebProxy = Nothing
  request.Proxy = System.Net.WebRequest.DefaultWebProxy

And changing the config file by adding

 <system.net>
   .......
   <connectionManagement>
     <add address="*" maxconnection="20"/>
   </connectionManagement>
 </system.net>

This still did not reduce the slow page request times on the server. In the end the solution was to uncheck the “Automatically detect settings” option in the IE options on the server itself. (Under Tools -> Internet Options select the Connections tab. Press the LAN Settings button)

Immediately after I unchecked this browser option on the server all the page request times dropped from 20+ seconds to under 1 second.

score 10 · Answer 4 · answered Jan 31 '14 at 10:20

I started observing a slow down similar to the OP in this area which got a little better when increasing the MaxConnections.

ServicePointManager.DefaultConnectionLimit = 4;

But after building this number of WebRequests the delays came back.

The problem, in my case, was that I was calling a POST and not bothered about the response so wasn't picking up or doing anything with it. Unfortunately this left the WebRequest floating around until they timed out.

The fix was to pick up the Response and just close it.

WebRequest webRequest = WebRequest.Create(sURL);
webRequest.Method = "POST";
webRequest.ContentLength = byteDataGZ.Length;
webRequest.Proxy = null;
using (var requestStream = webRequest.GetRequestStream())
{
    requestStream.WriteTimeout = 500;
    requestStream.Write(byteDataGZ, 0, byteDataGZ.Length);
    requestStream.Close();
}
// Get the response so that we don't leave this request hanging around
WebResponse response = webRequest.GetResponse();
response.Close();

Interesting. I had not thought of that because normally you also want to gather something from the response. I'm pretty sure this wasn't my actual problem, but interesting nonetheless — Earlz, Jan 31 '14 at 14:51
This didn't solve my problem but it lead me to review my use of `HttpWebResponse` and I found one that I was not closing. — Craig Silver, May 18 '18 at 19:48

score 3 · Answer 5 · answered Mar 25 '10 at 22:22

3

Use a computer other than localhost, then use WireShark to see what's really going over the wire.

Like others have said, it can be a number of things. Looking at things on the TCP level should give a clear picture.

answered Mar 25 '10 at 22:22

kervin

11,672
5
42
59

score 3 · Answer 6 · edited Jan 25 '12 at 19:52

3

I don't know how exactly I've reach to this workaround, I didn't have time to do some research yet, so it's up to you guys. There's a parameter and I've used it like this (at the constructor of my class, before instantiating the HTTPWebRequest object):

System.Net.ServicePointManager.Expect100Continue = false;

I don't know why exactly but now my calls look quite faster.

edited Jan 25 '12 at 19:52

animuson

53,861
28
137
147

answered Jan 25 '12 at 14:42

user753746

33
4

It can indeed be helpful if you POST / PUT (see [documentation](http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.expect100continue.aspx)). – Styxxy Nov 04 '13 at 22:24
It can also slow things down if there's a chance the server will reject the PUT / POST data (i.e. if you disable this, you won't know that the call was rejected until AFTER you PUT / POST the data). – BrainSlugs83 Jan 12 '14 at 04:24

score 1 · Answer 7 · answered Jun 11 '16 at 15:26

I tried all the solutions described here with no luck, the call took about 5 minutes.

What was the issue: I needed the same session and obviously the same cookies (request made on the same server), so I recreated the cookies from Request.Cookies into WebRequest.CookieContainer. The response time was about 5 minutes.

My solution: Commented out the cookie-related code and bam! Call took less then one second.

score 1 · Answer 8 · answered Oct 11 '17 at 10:06

1

I know this is some old thread, but I have lost whole day with slow HttpWebRequest, tried every provided solution with no luck. Every request for any address was more than one minute.

Eventually, problem was with my Antivirus Firewall (Eset). I'm using firewall with interactive mode, but Eset was somehow turned off completely. That caused request to last forever. After turning ON Eset, and executing request, prompt firewall message is shown, and after confirmation, request executing for less than one second.

answered Oct 11 '17 at 10:06

Šemsudin Tafilović

446
3
14

Thanks for the hint. I totally forgot about windows firewalls and it turns out, the first request can take eons when it's not configurated properly. Esp. When it is "turned off" in the win dialogs and the service is still running. – Xan-Kun Clark-Davis Jan 17 '18 at 20:03

score 1 · Answer 9 · answered Jun 17 '20 at 11:46

For me, using HttpWebRequest to call an API locally averaged 40ms, and to call an API on a server averaged 270ms. But calling them via Postman averaged 40ms on both environments. None of the solutions in this thread made any difference for me.

Then I found this article which mentioned the Nagle algorithm:

The Nagle algorithm increases network efficiency by decreasing the number of packets sent across the network. It accomplishes this by instituting a delay on the client of up to 200 milliseconds when small amounts of data are written to the network. The delay is a wait period for additional data that might be written. New data is added to the same packet.

Setting ServicePoint.UseNagleAlgorithm to false was the magic I needed, it made a huge difference and the performance on the server is now almost identical to local.

var webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ServicePoint.Expect100Continue = false;
webRequest.ServicePoint.UseNagleAlgorithm = false; // <<<this is the important bit

Note that this worked for me with small amounts of data, however if your request involves large data then it might be worth making this flag conditional depending on the size of the request/expected size of the response.

score 1 · Answer 10 · answered Nov 03 '21 at 18:37

We encountered a similar issue at work where we had two REST APIs communicating locally with each other. In our case all HttpWebRequest took more than 2 seconds even though the request url was http://localhost:5000/whatever which was way to slow.

However upon investigating this issue with Wireshark we found this:

It turned out the framework was trying to establish an IPv6 connection to localhost first, however as our services were listening on 0.0.0.0 (IPv4) the connection attempt failed and after 500 ms timeout it retried until after 4 failed attempts (4 * 500 ms = 2 seconds) it eventually gave up and used IPv4 as a fallback (which obviously succeeded almost immediately).

The solution for us was to change the request URI to http://127.0.0.1/5000/whatever (or to also listen on IPv6 which we deemed unnecessary for our local callbacks).

score 0 · Answer 11 · edited Sep 04 '19 at 20:12

0

This worked for me:

<configuration>
  <system.net>
    <defaultProxy enabled="false"/>
  </system.net>
</configuration>

Credit: Slow HTTPWebRequest the first time the program starts

edited Sep 04 '19 at 20:12

Ian Boyd

246,734
253
869
1,219

answered Dec 12 '16 at 12:01

mmmmmm

980
1
14
16

score 0 · Answer 12 · answered May 18 '18 at 23:03

For me, the problem was that I had installed LogMeIn Hamachi--ironically to remotely debug the same program that then started exhibiting this extreme slowness.

FYI, disabling the Hamachi network adapter was not enough because it seems that its Windows service re-enables the adapter.

Also, re-connecting to my Hamachi network did not solve the problem. Only disabling the adapter (by way of disabling the LogMeIn Hamachi Windows service) or, presumably, uninstalling Hamachi, fixed the problem for me.

Is it possible to ask HttpWebRequest to go out through a specific network adapter?

score 0 · Answer 13 · answered Nov 19 '20 at 10:47

In my case add AspxAutoDetectCookieSupport=1 to the code and the problem was solved

Uri target = new Uri("http://payroll");

string responseContent;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(target);

request.CookieContainer = new CookieContainer();         
request.CookieContainer.Add(new Cookie("AspxAutoDetectCookieSupport", "1") { Domain = target.Host });

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
    using (Stream responseStream = response.GetResponseStream())
    {
        using (StreamReader sr = new StreamReader(responseStream))
            responseContent = sr.ReadToEnd();
    }
}

score 0 · Answer 14 · answered Dec 15 '21 at 02:00

I was experiencing a 15 second or so delay upon creating a session to an api via httpwebrequest. The delay was around waiting for the getrequeststream() . After scrounging for answers, I found this article and the solution was just changing a local windows policy regarding ssl:

https://www.generacodice.com/en/articolo/1296839/httpwebrequest-15-second-delay-performance-issue

score -1 · Answer 15 · edited Mar 31 '16 at 09:14

-1

We had the same problem on web app. We waited on response 5 seconds. When we change user on applicationPool on IIS to networkService, the response began to arrive less than 1 second

edited Mar 31 '16 at 09:14

Community

1
1

answered Mar 31 '16 at 07:31

user2693593

9
1
2

HttpWebRequest is extremely slow!

15 Answers15

Linked