0

I am looking for a reliable and easy pattern to consume (execute) background tasks in .net core in parallel.

I found this https://stackoverflow.com/a/49814520/1448545 answer, but the problem is there is always single consumer of the tasks.

What if there is 1 new task to perform per 100ms, while each task takes 500ms to complete (e.g. long running API call). In such case the tasks will pile up.

How to make it dynamic, so if there is more items in BlockingCollection<TaskSettings> _tasks the .net core will create more tasks executors (consumer) dynamically?

Maciej Pszczolinski
  • 1,623
  • 1
  • 19
  • 38
  • `BlockingCollection` just contains the tasks pending for processing/executing. You determine how the tasks are executed (synchronously or asynchronously, sequential or parallel). So the problem turns into tasks scheduling. Also it may be more complicated if your tasks have some output that should be transferred to some next pipeline. – Hopeless Dec 26 '20 at 11:08
  • Thanks @Hopeless. I know it's only tasks pending. I want to find a pattern how to process them in parallel, how to spin up X tasks, and then kill these tasks. Looks like a common use case, but I haven't found anything, and I don't want to reinvent the wheel. – Maciej Pszczolinski Dec 26 '20 at 11:59
  • 1
    I'm pretty sure that someone has already implemented his own reusable lib (maybe exported to nuget) but there is one well-known lib owned by Microsoft named TPL Dataflow. You can use it in a hosted service run by asp.net core. Once using that you don't need your own BlockingCollection. Implementing the code your own is not too complicated to deal with some specific requirements but using a lib like TPL Dataflow can be better (in performance and supported scenarios) – Hopeless Dec 26 '20 at 13:14
  • I just checked TPL, it looks like one way is to have a singleton, that has one ActionBlock, and just Post data to that ActionBlock. It should also be called Complete() when app is about to terminate, and await Completion. Not sure though if that's a proper use of a TPL for my scenario. – Maciej Pszczolinski Dec 26 '20 at 14:41
  • Usually you don't need to call `Complete` unless you're sure that your block won't receive more tasks anymore. It's not necessary to care about it in this scenario, it's just like a waiting server living along with your web server – Hopeless Dec 26 '20 at 15:01
  • What I want to really do, is to process SQS messages in parallel, up to 100 in parallel. I think I can achieve it with action block and awaited SendAsync (this will block further SQS poll untill ActionBlock can accept more "work". How does it sound? – Maciej Pszczolinski Dec 26 '20 at 15:11
  • 1
    of course you can achieve that with TPL Dataflow (there is a max degree of parallelism option), you can find more about it (via documentation or example) – Hopeless Dec 26 '20 at 16:33

0 Answers0