0

I've a list of records that I need to distribute equally among limited no. of Tasks. enter image description here

Once assigned, those tasks should call following code for each record that is assigned to them and return each result simultaneously(updating Processed to N or Y and Status to Pass or Failed in real time) without waiting for all at once:

//Here we are calling an API with POST method and expecting a Tuple<message, StatusCode>
var pricingResult = await shippingService.PublishConfigShipping(apiRequest);
if (pricingResult.Item2 == System.Net.HttpStatusCode.OK)
   {
    //We are saving the data to sql db here
   result = await repository.SaveConfigShipping(data);
   }

I want to keep the tasks spawn limited so that API doesn't crash or data saving to DB doesn't slow down or crash. Can someone please let me know how to call this piece of code using Tasks?

LogicalDesk
  • 1,237
  • 4
  • 16
  • 46
  • 2
    If there's any possibility that different records might take different amounts of time to be processed, you can probably realise that partitioning the work into equal size sets *up front* might not be the best choice (especially e.g. if all of the slow ones get assigned to one task) – Damien_The_Unbeliever Jul 05 '22 at 13:14
  • 1
    API calls are executed asynchronously. Saving to the database is also performed asynchronously. There is no place for parallelism here. You should use the Producer-Consumer pattern. – Alexander Petrov Jul 05 '22 at 14:03
  • 1
    You can use `Parallel.ForEachAsync` to execute the requests and write to the database with a fixed DOP. Why would the database be affected by simple concurrent INSERTs though? That's usually a smell of inefficient data access code – Panagiotis Kanavos Jul 05 '22 at 14:35
  • 1
    What are you trying to do and what is the actual problem? It matters. Executing 100 concurrent INSERTs shouldn't be a problem - unless each operation reads and blocks rows for a long time. Quite often, inserting everything into a staging table and then processing the data at the end is far faster than working one row at a time, even if it's concurrently – Panagiotis Kanavos Jul 05 '22 at 14:37
  • 1
    What does `repository.SaveConfigShipping(data);` do? If all you want is to update a status, you just need to execute `UPDATE ThatTable SET Processed='Y',Status=@status WHERE RcID=@id`. That should be very fast even with thousand of concurrent operations, assuming `RcID` is the primary key, and no other connection is blocking rows. – Panagiotis Kanavos Jul 05 '22 at 14:46

1 Answers1

2

Have you considered something like Parallel.ForEach?

This would let the framework do the heavy lifting in determine the best way to split and schedule the work across available cores rather than implement all that yourself.

lee-m
  • 2,269
  • 17
  • 29