0

I want to convert a large array of objects in parallel using PLINQ.

var outputs = inputs
    .AsParallel()
    .AsOrdered()
    .WithDegreeOfParallelism(8) // make this number dynamic
    .Select(input => MyConverter(input))
    .ToArray();

Is there a way to change the number of threads used during execution (custom control)? E.g. by using a callback function like this:

int CurrentThreadLimit() => DateTime.Now.Second / 4 + 1; // callback function (example)

var outputs = inputs
    .AsParallel()
    .AsOrdered()
    .WithDynamicDegreeOfParallelism(() => CurrentThreadLimit()) // How to build this?
    .Select(input => MyConverter(input))
    .ToArray();

Is something like this possible?

Theodor Zoulias
  • 34,835
  • 7
  • 69
  • 104
stb
  • 772
  • 5
  • 15
  • Related: [Is it possible to change parallelOptions.MaxDegreeOfParallelism during execution of a Parallel.ForEach?](https://stackoverflow.com/questions/3705006/is-it-possible-to-change-paralleloptions-maxdegreeofparallelism-during-execution) – Theodor Zoulias Jun 16 '23 at 17:47
  • Already answered: https://stackoverflow.com/a/25290512/8978576 – Serhii Jun 16 '23 at 18:50
  • 1
    @Serhii I don't see a solution there for variable degree of parallelism during execution. – stb Jun 16 '23 at 21:17
  • Can't you make it a function that returns a degree of parallelism to a variable and then use this variable during a query? int degree = GetDegreeOfParallelism(); ...AsParallel().WithDegreeOfParallelism(degree); – Serhii Jun 18 '23 at 01:22
  • 1
    @Serhii the OP wants to change the degree of parallelism during a single execution of the query. For example start with parallelism 4, then change it to 2, and then change it back to 4 before the execution ends. They don't want a constant degree of parallelism from start to finish. – Theodor Zoulias Jun 18 '23 at 03:06
  • Actually, I don't know if something like this exists. Maybe, in such a scenario, you can use Load Balancing Partitioners for PLINQ (https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/custom-partitioners-for-plinq-and-tpl). Yeah, you can't pass exact number, but workload could be balanced between threads automatically. I hope this will help. – Serhii Jun 19 '23 at 23:04
  • @stb what's the *actual* problem and what kind of dynamic DOP are you looking for? The DOP is already dynamic, depending on the data source. PLINQ is meant for data parallelism. It works by partitioning the input into roughly as many partitions as there are cores and uses one worker per partition. There's no need to specify the DOP to begin with, unless you want to *reduce* it, so other processes on the same machine can still run. A *lot* depends on what `MyConverter` does. If performs IO or uses locks, execution speed will suffer – Panagiotis Kanavos Jun 20 '23 at 11:19
  • Yes, I want to limit processor use to something below Environment.ProcessorCount dynamically. MyConverter is using CPU (single threaded), no relevant locks or I/O. – stb Jun 20 '23 at 15:56

0 Answers0