Sequentially Splitting the load on a Parallel.Foreach Loop

Question

I have a million elements in a List to process.
Dropping them crudely into a Parallel.ForEach would just saturate the CPU.
Instead I split the Elements Master Lists into pieces and drop the Sublists into a parallel loop.

List<Element> MasterList = new List<Element>();
Populate(MasterList); // puts a Million elements into list;

//Split Master List into 100 Lists of 10.0000 elements each
List<List<Element>> ListOfSubLists = Split(MasterList,100);

foreach (List<Element> EL in ListOfSubLists )
{
   Parallel.ForEach(EL, E =>
   {
     // Do Stuff
   }

  //wait for all parallel iterations to end before continuing   
}

What is the best way for waiting for all parallel iteration to end before continuing to next iteration of the upper loop ?

Edit :
as some answers stated, "saturate the CPU" is not an accurate expression.
Actually I just want to limit the CPU usage, and avoid excessive load on it coming from this processing.

score 4 · Accepted Answer · edited May 23 '17 at 12:20

4

Parallel.ForEach will not saturate the CPU; it uses some intelligence to decide how many parallel threads to run simultaneously, with a max of 63.

See: Does Parallel.ForEach limits the number of active threads?

You can also set the max degree of parallelism if you want, by supplying a ParallelOptions like new ParallelOptions { MaxDegreeOfParallelism = 5 } as the second argument to Parallel.ForEach.

As a last point, Parallel.ForEach blocks until all of the iterations have completed. So your code, as written, works. You do not need to wait for the iterations to complete.

edited May 23 '17 at 12:20

Community

1
1

answered Feb 20 '12 at 14:16

David Pfeffer

38,869
30
127
202

2

I concur. Parallel.foreach enumerating 10,000 items won't create any more/less threads than a million. If RAM becomes an issue dealing with your million items, then consider BlockingCollection -- this will limit the number of items you have in memory at any one time. – Ragoczy Feb 20 '12 at 14:25
Thank you. As of the last sentence, I did not know that Parallel.ForEach was blocking. Very good to know. For CPU Saturation, See edit. – Mehdi LAMRANI Feb 20 '12 at 14:30
@MikaJacobi If you wanted nonblocking for some reason, you'd have to do it yourself by spawning new `Task`. – David Pfeffer Feb 20 '12 at 14:38
@Ragoczy: BlockingCollection is very interesting. It will be very useful to me indeed. A million thread is to much as the "Do Stuff" part is time-&-memory consuming that's why I wanted to limit it – Mehdi LAMRANI Feb 20 '12 at 14:41

score 2 · Answer 2 · answered Feb 20 '12 at 14:18

2

What do you mean "Saturate the CPU"

You can still throttle the Parallel foreach loop by supplying it with ParallelOptions One property of which is a MaxDegreesOfParallelism

That will allow you to go back to your single collection e.g.

Parallel.ForEach(
     collection,
     new ParallelOptions { MaxDegreeOfParallelism = 5},
     E => { DoStuff(E) }
);

answered Feb 20 '12 at 14:18

Eoin Campbell

43,500
17
101
157

Thank you, very useful to know. I was not aware of that. See edit. – Mehdi LAMRANI Feb 20 '12 at 14:31

Sequentially Splitting the load on a Parallel.Foreach Loop

2 Answers2

Linked