With PLINQ, how can I set the preferred chunk size when using AsParallel().MaxDegreeOfParallelism(4)?

Question

I have a list with thousands of objects, on which an operation that can take between 1 and 3 minutes is performed. I am of course using PLINQ, but I have noticed that when approaching the end of the input list, only one core is working, like if the partitioning had been determined ex ante.

So, with a IList, what is the best way to force PLINQ to keep using worker threads as long as there are items to be processed? The computer has plenty of hardware cores available.

References:

• Chunk partitioning vs range partitioning in PLINQ
• Can I configure the number of threads used by Parallel Extensions?
• How to make PLINQ to spawn more concurrent threads in .NET 4.0 beta 2?

Answer 1

From what I understand, PLINQ will choose range or chunk partitioning depending on whether the source sequence is an IList or not. If it is an IList, the bounds are known and elements can be accessed by index, so PLINQ chooses range partitioning to divide the list evenly between threads. For instance, if you have 1000 items in your list and you use 4 threads, each thread will have 250 items to process. On the other hand, if the source sequence is not an IList, PLINQ can't use range partitioning because it doesn't know what the ranges would be; so it uses chunk partitioning instead.

In your case, if you have an IList and you want to force chunk partitioning, you can just make it look like a simple IEnumerable: instead of writing this:

list.AsParallel()...

Write that:

list.Select(x => x).AsParallel()...

The dummy projection will hide the fact that the source is actually an IList, so PLINQ will use chunk partitioning.