2

I have a .net core api that must make around 150,000 calls to collect data from external services. I am running these requests in parallel using Parallel.forEach and that seems to be working great, however I get an error from the http client for around 100,000 of my requests!

The Operation was canceled

Looking back at this I wish I had also logged the exception type but I believe this is due to not having enough outgoing connection limit.

Through debugging I have found that this returns 2:

ServicePointManager.DefaultConnectionLimit

On the face of it, if this really is the maximum amount of open connections allowed to an external domain / server, I want to increase that as high as possible. Ideally to 150,000 to ensure parallel processing doesnt cause an issue.

The problem is I cant find any information on what a safe limit is, or how much load this will put on my machine - if it is even a lot. Since this issue causes a real request to be made my data provider counts it in my charges - but obviously I get nothing from it since the .net core framework is just throwing my result away..

Since this problem is also intermittent it can be difficult to debug and I would just like to set this value as high as is safe to do so on my local machine.

I believe this question is relevant to stackoverflow since it does deal directly with the technical issue above, whereas other questions I could find only ask details about what this setting is.

Craig
  • 1,648
  • 1
  • 20
  • 45
  • Suggestion: create some test benchmarks and stress tests on your workstation (i.e. localhost). Try to get "quantitive data" in your test environment first. Look here: https://learn.microsoft.com/en-us/aspnet/core/test/load-tests?view=aspnetcore-2.2 – paulsm4 Feb 17 '19 at 22:32
  • Parallel.ForEach is meant for CPU intensive operations, not I/O intensive operations like you have. You'll want Task.WhenAll instead, possibly with a semaphore, to control the number of open connections, like this example: https://stackoverflow.com/a/37569259/94853 – Loren Paulsen Feb 17 '19 at 22:36
  • Paulsen I went that route at first as well but from what I read task. When All still keeps the process on 1 logical core. I want to speed this up to run on all cores. Please correct me if I'm wrong though... – Craig Feb 17 '19 at 22:38
  • I would be concerned that network I/O will be your bottleneck even with one core. Multiple cores would just exhaust all available ports faster, and may actually reduce the performance by adding more overhead. In practice, I've limited to 50 open connections before, but that was just one application. This is something you'll want to test for yourself, but it's unlikely to benefit from being thousands. – Loren Paulsen Feb 17 '19 at 22:41
  • `The Operation was canceled` seems more like a timeout from an external service than anything else. – JohanP Feb 17 '19 at 23:04
  • timeouts are logging as a different error for me JohanP - the message is something like 'took too long to respond'. I should be logging my exception type and stack trace I suppose. – Craig Feb 17 '19 at 23:05
  • @Craig using httpClient and doing an async call, how do you detect a timeout then? There is no `Timeout` property on the `Response`object, it's usually in the form of the operation was canceled. – JohanP Feb 17 '19 at 23:08
  • @Craig How are you using your `HttpClient`? You injecting an `IHttpClientFactory`? I had an app that was making up to 30million+ requests a day and I had no issues with ServicePointManager – JohanP Feb 17 '19 at 23:13
  • I create httpclient as a singleton at app startup and add it to the di container. – Craig Feb 18 '19 at 00:00
  • cant really comment on the other point other than that i have logs where the log message is literally - they took too long to respond - therefore this cant be the same issue. – Craig Feb 18 '19 at 00:01

1 Answers1

4

As far as I understand, you are trying to make 150000 simulatenous request to external services. I presume that your services are Restful web services. If that is the case when you set DefaultConnectionLimit to an arbitrary number (very high), every single request opens a port for requesting data. This definitely clogs your network and your ports (port range is 0 to 65535).

Besides, making 150000 request without using throttling uncontrollably consumes your OS resources.

DefaultConnectionLimit is there because it protects you from aforementioned problems.

you may consider to use SemaphoreSlim for throttling

Derviş Kayımbaşıoğlu
  • 28,492
  • 4
  • 50
  • 72
  • I guess the idea of implementing a semaphore is that it becomes impossible to go over my outgoing limit if one thread = 1 request pipeline. Sounds like a good idea to me – Craig Feb 17 '19 at 23:04
  • this was a great specific answer - for any other brainless people out there like me though; my issue is that i should not be doing this to begin with. run batches and block next batch until previous finishes or it will shred through resources and have the issues I have had here. – Craig Feb 18 '19 at 09:59