2

We got a requirement to execute bulk API request(more than 1000 calls) and save response data in database.

Initially we got 10-20 API calls requirement. For these requests we used simple 'for loop' with single instance of HTTPClient to fetched data and saved it to database.

Later we got 100-200 API calls requirement. For these requests we used Parallel.ForEach with single HTTPClient instance but we are getting socket issues. so we created new instance for each request. even this approach causing issue on server .

Now requirement is to execute 1000+ API calls.

With single HTTPClient instance using 'await' taking hours to complete the operation.

Is there any way or good practice for doing 1000+ api call with very good performance and without loosing data in C#?

Sai Kumar
  • 308
  • 2
  • 10
  • Did you try firing off the async client calls without the await and use `Task.WhenAll`? Also, is there any chance you control the endpoint you're calling? If you do then you might consider making a more bulk friendly endpoint where you can pass collections of parameters and return a collection of responses to cut down in HTTP traffic if that approach works for your scenario. – knowonecanknow Dec 02 '20 at 18:06
  • 2
    Assuming you're not using net core (and could use IHttpClientFactory) see [You're using HttpClient wrong and it is destabilizing your software](https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/) - that (probably) explains why you saw socket issues, and how to resolve them. – stuartd Dec 02 '20 at 18:06
  • @stuartd- I went through that link but by using await its taking hours to complete the operation. – Sai Kumar Dec 02 '20 at 18:10
  • 1
    Have you set a connection limit on the API Uri - `ServicePointManager.FindServicePoint(new Uri(url)).ConnectionLimit = howeverManyConcurrentRequestsYouWant;` or the [ServicePointManager.DefaultConnectionLimit](https://learn.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.defaultconnectionlimit)? What's `netstat` showing for connections? – stuartd Dec 02 '20 at 18:17
  • 2
    See [How to make concurrent requests with HttpClient](https://makolyte.com/csharp-how-to-make-concurrent-requests-with-httpclient/) for more detail – stuartd Dec 02 '20 at 18:19
  • @knowonecanknow - we dont have chance to control endpoint. Yes we used Task.WhenAll but it not worked. – Sai Kumar Dec 02 '20 at 18:20
  • Try to use when all in batches. Since for every one of the calls you are also doing database work, it might well be that you are causing bottlenecks in your database or backend server. So make 20 calls in parallel, then another twenty and so forth. – Athanasios Kataras Dec 02 '20 at 18:25
  • @AthanasiosKataras- currently I am running in batches and doing merge in database. But I am searching for best approach for doing bulk API calls from C# which can complete as soon as possible. – Sai Kumar Dec 02 '20 at 18:34
  • [Throttling asynchronous tasks](https://stackoverflow.com/q/22492383/7444103) -- [Queue of async tasks with throttling which supports muti-threading](https://stackoverflow.com/a/34316994/7444103) – Jimi Dec 02 '20 at 19:07
  • @AthanasiosKataras- Max how many HTTPClient instances we can create and how requests can one HTTPClient instance will serve – Sai Kumar Dec 03 '20 at 06:31
  • https://stackoverflow.com/questions/39516598/maximum-concurrent-requests-for-webclient-httpwebrequest-and-httpclient – Sai Kumar Dec 03 '20 at 10:53

1 Answers1

0

You can try a combination of the solution in the comments with the when all: https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.whenall?view=net-5.0

private static HttpClient Client = new HttpClient();
public static async Task Main(string[] args) 
{
    var tasks = new List<Task>();      
    Console.WriteLine("Starting connections");
    for(int i = 0; i<10; i++)
    {
        tasks.Add(Client.GetAsync("http://aspnetmonsters.com"))
        Console.WriteLine(result.StatusCode);
    }
    Task t = Task.WhenAll(tasks);
    try {
       t.Wait();
    }
    catch {}   
    Console.WriteLine("Connections done");
    Console.ReadLine();
}
Athanasios Kataras
  • 25,191
  • 4
  • 32
  • 61