Is ServicePointManager.DefaultConnectionLimit by IP?

Question

I'm building a web crawler and the target site does not allow more than 2 concurrent connections from the same IP.

My plan was: I would buy an EC2 with 4 additional elastic ip's to get data faster (would be 10 concurrent connections, 8 from additional IP's and 2 from the "main" IP).

I would use the BindIPEndPointDelegate (took from here) to set the source IP and then start download the pages.

So here my question comes in: The DefaultConnectionLimit will be applied to each source IP or will be the entire application limited to 2 concurrent connections?

Technically, it is per ServicePoint, which effectively should be the same as per IP. — Crowcoder, Nov 12 '17 at 17:45

score 0 · Answer 1 · answered Oct 27 '18 at 22:47

You need to specify the host otherwise it applies to all hosts:

Using config:

<!-- By host -->
<configuration>
 <system.net>
  <connectionManagement>
   <add address="myapi.com" maxconnection="12"/>
   <add address="yourapi.com" maxconnection="8"/>
   <add address="74.125.236.199" maxconnection="4"/>
  </connectionManagement>
 </system.net>
</configuration>

<!-- All hosts -->
<configuration>
 <system.net>
  <connectionManagement>
   <add address="*" maxconnection="15"/>
  </connectionManagement>
 </system.net>
</configuration>

Using code:

var apiServicePoint1 = ServicePointManager.FindServicePoint("myapi.com");

apiServicePoint1.ConnectionLimit = 12;

var apiServicePoint2 = ServicePointManager.FindServicePoint("yourapi.com");

apiServicePoint2.ConnectionLimit = 8;

Reference: https://vincentlauzon.com/2015/12/13/beyond-2-concurrent-connections-in-net/

Is ServicePointManager.DefaultConnectionLimit by IP?

1 Answers1