92

I have an ASP.NET 4.0 application running atop IIS 7.5 on a 64-bit Windows Server 2008 R2 Enterprise machine with gobs of RAM, CPU, disk, etc.

With every web request, the ASP.NET application makes a connection to a backend web service (via raw sockets), which is running on the same machine.

Problem: There appears to be something limiting the # of simultaneous connections to the backend web service. Suspiciously, the number of concurrent connections is topping out at 16.

I found this key article from Microsoft explaining how to tweak IIS' settings to accomodate ASP.NET apps that make lots of web service requests: http://support.microsoft.com/?id=821268#tocHeadRef

I followed the article's recommendatinos, but still no luck. The setting that is particularly interesting is the maxconnection setting, which I even bumped to 999.

Any ideas what else could be throttling connections?

Note: When I cut IIS out of the mix and have the clients connect directly to the backend web service, it will happily open as many connections as I need, so I'm positive the backend is not the bottleneck. It must be something in IIS/ASP.NET-land.

Here's the relevant section of the machine.config which I'm sure is being read by the application (verified with appcmd.exe):

<system.web>
    <processModel autoConfig="false" maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" />
    <httpRuntime minFreeThreads="176" minLocalRequestFreeThreads="152"/>

    <httpHandlers />

    <membership>
        <providers>
            <add name="AspNetSqlMembershipProvider"
                type="System.Web.Security.SqlMembershipProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                connectionStringName="LocalSqlServer"
                enablePasswordRetrieval="false"
                enablePasswordReset="true"
                requiresQuestionAndAnswer="true"
                applicationName="/"
                requiresUniqueEmail="false"
                passwordFormat="Hashed"
                maxInvalidPasswordAttempts="5"
                minRequiredPasswordLength="7"
                minRequiredNonalphanumericCharacters="1"
                passwordAttemptWindow="10"
                passwordStrengthRegularExpression="" />
        </providers>
    </membership>

    <profile>
        <providers>
            <add name="AspNetSqlProfileProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Profile.SqlProfileProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </profile>

    <roleManager>
        <providers>
            <add name="AspNetSqlRoleProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Security.SqlRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add name="AspNetWindowsTokenRoleProvider" applicationName="/"
                type="System.Web.Security.WindowsTokenRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </roleManager>
</system.web>
<system.net>
    <connectionManagement>
        <add address="*" maxconnection="999"/>
    </connectionManagement>
</system.net>
Rob Sobers
  • 20,737
  • 24
  • 82
  • 111
  • @DanB has a good point here - how are you measuring the number of concurrent connections? – Jeremy McGee Oct 24 '11 at 13:57
  • @JeremyMcGee I'm measuring the # of concurrent connections by running TCPView on the server to see how many backend connections are made by the IIS worker processes. – Rob Sobers Oct 24 '11 at 14:34
  • Are the Web clients running on independent machines or the same machine? (Checking that there's no client-side throttling going on.) – Jeremy McGee Oct 24 '11 at 14:49
  • 3
    @JeremyMcGee Separate machines. 1 client-server connection per machine. Also, we know there isn't any throttling somewhere on the network because when we hit the backend directly (which happens over HTTP as well) we don't run into the bottleneck. – Rob Sobers Oct 24 '11 at 15:02
  • We had this same issue with a console app that called a webservice and needed to do lots of simultaneous calls. We discovered the console app was throttling itself to 2 connections. Setting System.Net.ServicePointManager.DefaultConnectionLimit = 1000; cleared the issue, but now we're using the same code on a website and I'm wondering how this setting relates to the solutions talked about here. Which one takes priority? – Glenn Jan 08 '13 at 19:11
  • 1
    Rob, did you ever find a definitive solution to this? – electronicKT Apr 10 '13 at 11:48
  • Neither solution using maxconnection="65535" ? In ASP.NET 4.5 new configuration sections maybe ? – Kiquenet Oct 21 '14 at 19:06

7 Answers7

106

Most of the answers provided here address the number of incoming requests to your backend webservice, not the number of outgoing requests you can make from your ASP.net application to your backend service.

It's not your backend webservice that is throttling your request rate here, it is the number of open connections your calling application is willing to establish to the same endpoint (same URL).

You can remove this limitation by adding the following configuration section to your machine.config file:

<configuration>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="65535"/>
    </connectionManagement>
  </system.net>
</configuration>

You could of course pick a more reasonable number if you'd like such as 50 or 100 concurrent connections. But the above will open it right up to max. You can also specify a specific address for the open limit rule above rather than the '*' which indicates all addresses.

MSDN Documentation for System.Net.connectionManagement

Another Great Resource for understanding ConnectManagement in .NET

Hope this solves your problem!

EDIT: Oops, I do see you have the connection management mentioned in your code above. I will leave my above info as it is relevant for future enquirers with the same problem. However, please note there are currently 4 different machine.config files on most up to date servers!

There is .NET Framework v2 running under both 32-bit and 64-bit as well as .NET Framework v4 also running under both 32-bit and 64-bit. Depending on your chosen settings for your application pool you could be using any one of these 4 different machine.config files! Please check all 4 machine.config files typically located here:

  • C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG
  • C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG
  • C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config
  • C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config
BenSwayne
  • 16,810
  • 3
  • 58
  • 75
  • Yup, I covered that base. Thanks anyway, @BenSwayne! I verified that I modified the correct `machine.config` (64-bit 4.0). – Rob Sobers Oct 26 '11 at 13:41
  • @RobSobers: In this case I would be a little suspicious of the implementation code. Perhaps you are running out of threads or something? Can you throw your TcpClient webservice call code into a console app and see if you can achieve a better request rate? This will prove whether its an IIS specific configuration or a broader .NET configuration or your code. – BenSwayne Oct 26 '11 at 23:49
  • Does this line go in the client machine? – Uri Abramson Jun 10 '13 at 09:04
  • thank you , your settings combine with processModel twekaing from http://www.codeproject.com/Articles/133738/Quick-Ways-to-Boost-Performance-and-Scalability-of fixed "ISAPI ‘C:\windows\Microsoft.Net\Framework\v2.0.050727\aspnet_isapi.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected" problem – Zakos Jan 14 '15 at 10:13
  • @BenSwayne does the same concept applies for SMTP connections as well? I am designing a bulk email sending application so will this **ConnectionManagement** property will be useful in sending bulk mails as well (using multi threading for **mail.send** function)? – vibs2006 Dec 27 '16 at 16:50
8

I realize the question might be rather old, but you say the backend is running on the same server. That means on a different port, probably other than the default port 80.

I've read that when you use the "connectionManagement" configuration element, you need to specify the port number if it differs from the default 80.

LINK: maxConnection setting may not work even autoConfig = false in ASP.NET

Secondly, if you choose to use the default configuration (address="*") extended with your own backend specific value, you might consider putting the specific value first! Otherwise, if a request is made, the * matches first and the default of 2 connections is taken. Just like when you use the section in web.config.

LINK: <remove> Element for connectionManagement (Network Settings)

Hope it helps someone.

Sebastian Hek
  • 81
  • 1
  • 3
6

Might it be possible that you're using a WCF-based web service reference? By default, the ServiceThrottlingBehavior.MaxConcurrentCalls is 16.

You could try updating your service reference behavior's <serviceThrottling> element

<serviceThrottling
    maxConcurrentCalls="999" 
    maxConcurrentSessions="999" 
    maxConcurrentInstances="999" />

(Not that I'd recommend the settings above.) See MSDN for more information how to configure an appropriate <behavior> element.

Ruben
  • 15,217
  • 2
  • 35
  • 45
  • I wish we were, but we're not. The service being consumed is running in Apache/Python/mod_wsgi on another machine, and, as Rob mentioned, it's clearly not the issue. – Benjamin Pollack Oct 24 '11 at 18:15
  • Service References are on the client. How is your client consuming the Apache service? – John Saunders Oct 24 '11 at 19:08
  • Like John said: the throttling is set on the client consuming the web service. Perhaps the phrase "your service behavior" is a bit misleading. Does it make more sense when phrased as "your service reference behavior"? – Ruben Oct 24 '11 at 21:17
  • @john It's consumed via a `TcpClient` (the service vends custom binary blobs, not WCF/SOAP or similar). Those configuration options *seem* to be purely impacting you if you use `ServiceHost`, not `TcpClient`; am I missing something? – Benjamin Pollack Oct 26 '11 at 14:46
  • Why not use "Add Service Reference"? – John Saunders Oct 26 '11 at 15:43
3

Have you tried to set the value of the static DefaultConnectionLimit property programmatically?

Here is a good source of information about that true headache... ASP.NET Thread Usage on IIS 7.5, IIS 7.0, and IIS 6.0, with updates for framework 4.0.

Simon Mourier
  • 132,049
  • 21
  • 248
  • 298
  • I imagine this should not matter, but I'm going to try it anyway. – Rob Sobers Oct 26 '11 at 13:41
  • This is how we fixed it a few years back. We had concurrency issues and this seemed to clear it up. Net.ServicePointManager.DefaultConnectionLimit = 1000 is what we used. You must set it BEFORE you create connection classes. – Brain2000 Jul 06 '17 at 17:49
2

See the "Threading" section of this page: http://msdn.microsoft.com/en-us/library/ff647786.aspx, in conjunction with the "Connections" section.

Have you tried upping the maxconnection attribute of your processModel setting?

Matt Evans
  • 7,113
  • 7
  • 32
  • 64
  • Yes, I have. But that article you cited points out something interesting that ever other article on the topic neglected to mention: that the `maxconnection` attribute does not apply to **local** web service calls. In this particular case the web service *is* local, but we have that same problem in another environment where the service is not local. – Rob Sobers Oct 24 '11 at 16:47
  • @RobSobers - I think that the above link is worthy of a second look. Check the part about `minLocalRequestFreeThreads` - **This worker process uses this setting to queue up requests from localhost (where a Web application _calls a Web service on the same server_) if the number of available threads in the thread pool falls below this number. This setting is similar to minFreeThreads, but it only _applies to requests that use localhost_.** – Ahmad Oct 26 '11 at 05:15
  • @Ahmad Yup, I set `minLocalRequestFreeThreads` as well (see my `machine.config` in the question. – Rob Sobers Oct 26 '11 at 13:43
0

If it is not defined in the web service or application or server (apache or IIS) that is hosting the web service consumable then you could create infinite connections until failure

Frank Tudor
  • 4,226
  • 2
  • 23
  • 43
0

while doing performance testing, the measure i go by is RPS, that is how many requests per second can the server serve within acceptable latency.

theoretically one server can only run as many requests concurrently as number of cores on it..

It doesn't look like the problem is ASP.net's threading model, since it can potentially serve thousands of rps. It seems like the problem might be your application. Are you using any synchronization primitives ?

also whats the latency on your web services, are they very quick to respond (within microseconds), if not then you might want to consider asynchronous calls, so you dont end up blocking

If this doesnt yeild something, then you might want to profile your code using visual studio or redgate profiler

np-hard
  • 5,725
  • 6
  • 52
  • 76