1

I have a ASP.NET Web Forms application that internally makes many SOAP and REST calls to web services. The SOAP calls are made using Microsoft's own "wrapping" code. The REST calls are made from a simple REST client. This client uses Using blocks to dispose of any resources.

The application runs fine for a few hours but then hangs up. The site will no longer be able to serve up asp.net pages. I tested the site by putting a hello.htm file on it... it served up fine. So it is definitely a problem in the ASP.NET engine space.

It feels like it has run out of webrequests from a pool. I could be way off. What can I try? What should I look at? It takes hours to get the problem to repro. Restarting the site on IIS fixes the issue, but of cource is not really a fix we can live with.

start of error and stack trace:
Server Error in '/' Application.

The operation has timed out Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.WebException: The operation has timed out

Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace: [WebException: The operation has timed out] System.Net.HttpWebRequest.GetRequestStream() +5322142 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) +103

UPDATE and MORE INFO
The homegrown REST calls and SOAP are all done synchronous.
They are not long-lived they take about 1 sec.
Web Services are hosted on a Tomcat server (different machine) in the same data center.
The .NET app calls SOAP and REST services on the Tomcat server.

The app somehow fixed itself after about an hour of staying in this hung state. Thoughts? How do I monitor the .NET Threadpool? Does DefaultConnectionLimit affect the self-initiated outbound connections?

RESOLUTION see additional comments to answers and... 1) close all streams and responses/requests from your homegrown REST code (oops)
2) System.Net.ServicePointManager.DefaultConnectionLimit = 96;
// default = 12 times # of cores
System.Net.ServicePointManager.MaxServicePointIdleTime= 3000;
// default = 100000 (100 seconds)
By fixing any leaky code and timing out the IdleTime faster everything seems to work fine now

BuddyJoe
  • 69,735
  • 114
  • 291
  • 466
  • Are there any usage limits on the external SOAP services you are calling? Track down which service it's calling and see if you can hit it repeatedly as many times are you are hitting it in practice. Do all pages call this same service? Do all SOAP calls fail or just this one? – Ian Mercer Jul 26 '10 at 19:52
  • I can hit it from the same box with a WinForms client and another ASP.NET web app (same IIS). All calls REST or SOAP from this IIS site timeout. The Applications log has it listed as a WebException Timeout – BuddyJoe Jul 26 '10 at 20:02
  • How do I check how much memory this app is taking up? I'm not yet that familiar with navigating IIS 7 or win2008 – BuddyJoe Jul 26 '10 at 20:05
  • Does anyone know if .NET 4.0 on IIS 7 would handle the creation and destruction of lots of WebRequests better than .NET 3.5 on IIS 7? – BuddyJoe Jul 27 '10 at 13:51

1 Answers1

3

1) Are you using asynchronous WebRequests? 2) Are your web service calls long lived (meaning they take a long time to complete?) 3) Where is the webservice hosted? Is it on the same machine? 4) Where is the REST service hosted? Same machine as asp.net app or different?

When a request comes into an ASP.NEt server, it is handled on a thread pool thread. I am not sure if this will be a completion port thread or a thread pool thread.

Anyway, your app is being invoked on a threadpool thread. On this thread you are making an outbound HTTPWebRequest (HWR). If this request is synchronous, it should not take any extra thread. However if it is asynchronous it will take another thread from the pool to complete.

Now, if the HWR you made is back to the same server, and that request needs another thread on the same server to complete, you now have a requirement for one more thread. Combine this with the Webservice taking a long time, and you are tying up atleast 2 (max 3) threads per request. If you are issuing a lot of request to the asp.net server, the server can quickly hit up against the max threadpool thread limit.

So far, I am assuming that there are no logic bugs in your code, and you are disposing of your HttpWebResponse objects correctly.

So, when there is no thread to finish the work, your HttpWebRequest can throw a timeout exception.

This might also happen if your backend webservice calls are to the same machine, and you have not increased the ServicePointManager.DefaultConnectionLimit to a reasonable value for your scenario. For eg, if your asp.net app expects to be hit with 200 simultaneous requests, you should set the DefaultConnectionLimit = 200 + some reasonable buffer. Lack of connection in the connectiion pool can also cause this.

Solution:

I would first up the DefaultConnectionLimit. If that doesnt solve it, then you would need to monitor ASP.NET performance counters and see whether the .Net threadpool is being exhausted which could contribute to webrequests failing.

feroze
  • 7,380
  • 7
  • 40
  • 57
  • 1
    Yes, DefaultConnectionLimit affects the outbound connections initiated from your application. As a first test, I would try increasing that to see if it fixes your problem. For monitoring thread count you can see the performance counter .NET CLR Locks and threads/# of current physical threads as a proxy for the # threads in the threadpool. You can also track asp.net perf counters which track requests queued/requests executing etc – feroze Jul 28 '10 at 20:30
  • Called Microsoft support on this one and these 2 settings were key- http://stackoverflow.com/questions/3363183 – BuddyJoe Jul 30 '10 at 13:09