Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)

Question

I'm developing an application (winforms C# .NET 4.0) where I access a lookup functionality from a 3rd party through a simple HTTP request. I call an url with a parameter, and in return I get a small string with the result of the lookup. Simple enough.

The challenge is however, that I have to do lots of these lookups (a couple of thousands), and I would like to limit the time needed. Therefore I would like to run requests in parallel (say 10-20). I use a ThreadPool to do this, and the short version of my code looks like this:

public void startAsyncLookup(Action<LookupResult> returnLookupResult)
{
    this.returnLookupResult = returnLookupResult;

    foreach (string number in numbersToLookup)
    {
        ThreadPool.QueueUserWorkItem(lookupNumber, number);
    }
}

public void lookupNumber(Object threadContext)
{
    string numberToLookup = (string)threadContext;
    string url = @"http://some.url.com/?number=" + numberToLookup;
    WebClient webClient = new WebClient();
    Stream responseData = webClient.OpenRead(url);
    LookupResult lookupResult = parseLookupResult(responseData);

    returnLookupResult(lookupResult);
}

I fill up numbersToLookup (a List<String>) from another place, call startAsyncLookup and provide it with a call-back function returnLookupResult to return each result. This works, but I found that I'm not getting the throughput I want.

Initially I thought it might be the 3rd party having a poor system on their end, but I excluded this by trying to run the same code from two different machines at the same time. Each of the two took as long as one did alone, so I could rule out that one.

A colleague then tipped me that this might be a limitation in Windows. I googled a bit, and found amongst others this post saying that by default Windows limits the number of simultaneous request to the same web server to 4 for HTTP 1.0 and to 2 for HTTP 1.1 (for HTTP 1.1 this is actually according to the specification (RFC2068)).

The same post referred to above also provided a way to increase these limits. By adding two registry values to [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings] (MaxConnectionsPerServer and MaxConnectionsPer1_0Server), I could control this myself.

So, I tried this (sat both to 20), restarted my computer, and tried to run my program again. Sadly though, it didn't seem to help any. I also kept an eye on the Resource Monitor while running my batch lookup, and I noticed that my application (the one with the title blacked out) still only was using two TCP connections.

So, the question is, why isn't this working? Is the post I linked to using the wrong registry values? Is this perhaps not possible to "hack" in Windows any longer (I'm on Windows 7)?

And just in case anyone should wonder, I have also tried with different settings for MaxThreads on ThreadPool (everything from 10 to 100), and this didn't seem to affect my throughput at all, so the problem shouldn't be there either.

Do we know why there are two keys FEATURE_MAXCONNECTIONSPERSERVER and FEATURE_MAXCONNECTIONSPER1_0SERVER? What is the significance of these two separate keys? Logically there should have been only one. — RBT, Jun 19 '16 at 02:56

score 96 · Accepted Answer · answered Jun 02 '10 at 17:45

96

It is matter of ServicePoint. Which provides connection management for HTTP connections. The default maximum number of concurrent connections allowed by a ServicePoint object is 2. So if you need to increase it you can use ServicePointManager.DefaultConnectionLimit property. Just check the link in MSDN there you can see a sample. And set the value you need.

answered Jun 02 '10 at 17:45

Incognito

16,567
9
52
74

Thanks to both of you for great question and answer – nikib3ro Jun 21 '11 at 17:07
If we set the defaultconnectionlimit property on servicepointmanager would that be enough or do we need to update the above registry settings as well? – Rajesh Feb 27 '14 at 11:07
2

@Rajesh No need to update registry, just setting the property value would be enough – Incognito Feb 27 '14 at 12:55
One thing to note is there is a reason for the default limit to be so low. Essentially to be a good internet citizen. If this is hitting your own services increasing the number is obviously not a problem as you can control the load. For url's that aren't yours, you should contact the owner before hand or be ok with the endpoint you are calling throttling you anyway. This is unfortunately a way to do simple DOS attacks on small and/or poorly written sites even when that is not your intention. – btlog Apr 21 '15 at 19:04
@Nailuj what was the maximum parallelism that you were able to achieve after using this service point settings change. Was it 10,20, 100 or anywhere in between which was of your choice? – RBT Jun 19 '16 at 02:46

score 29 · Answer 2 · answered Oct 23 '12 at 06:29

For quicker reference for someone. To increase the connection limit per host you can do this in your Main() or anytime before you begin making the HTTP requests.

   System.Net.ServicePointManager.DefaultConnectionLimit = 1000; //or some other number > 4

score 2 · Answer 3 · edited Apr 21 '15 at 18:47

Fire and forget this method from your main method. Icognito user is correct, only 2 threads are allowed to play at the same time.

private static void openServicePoint()
{
    ServicePointManager.UseNagleAlgorithm = true;
    ServicePointManager.Expect100Continue = true;
    ServicePointManager.CheckCertificateRevocationList = true;
    ServicePointManager.DefaultConnectionLimit = 10000;
    Uri MS = new Uri("http://My awesome web site");
    ServicePoint servicePoint = ServicePointManager.FindServicePoint(MS);
}

score 0 · Answer 4 · answered Jun 02 '10 at 17:41

0

For Internet Explorer 8: Run Registry Editor and navigate to following key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTION SPERSERVER

and

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTION SPER1_0SERVER

If FEATURE_MAXCONNECTIONSPERSERVER and FEATURE_MAXCONNECTIONSPER1_0SERVER are missing then create them. Now create DWORD Value called iexplore.exe for both sub keys (listed above) and set their value to 10 or whatever number desired.

answered Jun 02 '10 at 17:41

Gary

2,866
1
17
20

I can't seem why this would work, as this only throttles the limits for IE, and not in general... Anywho, Incognito's answer solved my problem. – Julian Jun 02 '10 at 17:53
It should not be iexplore.exe but the name of your process executable and limit only applies to connections going out of WebBrowser control. – Alex Pakka May 13 '14 at 20:06

Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)

4 Answers4

Linked

Related