23

Why Parallel.ForEach loop exits with OperationCancelledException, while using GetConsumableEnumerable?

//outside the function
static BlockingCollection<double> _collection = new BlockingCollection<double>();
    
    
var t = Task.Factory.StartNew(Producer);            
Parallel.ForEach(_collection.GetConsumingEnumerable(),
    item => Console.WriteLine("Processed {0}", item));
Console.WriteLine("FINISHED processing");


public static void Producer()
{
     var data = Enumerable.Range(1, 1000);
     foreach (var i in data)
     {
        _collection.Add(i);
        Console.WriteLine("Added {0}",i);
     }
                    
     Console.WriteLine("Finished adding");
     _collection.CompleteAdding();
}
Theodor Zoulias
  • 34,835
  • 7
  • 69
  • 104
Sam
  • 231
  • 2
  • 4
  • I can't reproduce the `OperationCancelledException` behavior on .NET 7. The code in the question runs successfully to completion, no exception is thrown. – Theodor Zoulias Nov 24 '22 at 09:36
  • It should be noted that when using a `BlockingCollection` as the source of a parallel operation, it is recommended to use a `Partitioner` configured with the `EnumerablePartitionerOptions.NoBuffering` option, as shown [here](https://stackoverflow.com/questions/10208330/why-does-iterating-over-getconsumingenumerable-not-fully-empty-the-underlying/33901522#33901522). Otherwise the consumer might try to bite more than it can chew, resulting in increased latency, and potentially in a deadlock. – Theodor Zoulias Nov 24 '22 at 09:54

1 Answers1

25

Using Parallel.ForEach with BlockingCollection is somewhat problematic, as I found out recently. It can be made to work, but it needs a little extra effort.

Stephen Toub has an excellent blog post on it, and if you download the "Parallel Extension Extras" project (also available on NuGet) you'll find some code ready to help you.

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
  • What really puzzles me is why Parallel.ForEach throws exception when I call _collection.CompleteAdding(). – Sam Jul 07 '11 at 09:52
  • @Sam: I wouldn't like to say, to be honest. There's too much deep magic going on there for me to have any confidence in saying the right thing :) – Jon Skeet Jul 07 '11 at 10:01
  • The current URL to the Parallel Extensions Extras: https://code.msdn.microsoft.com/ParExtSamples and someone has made a NuGet of the extensions: https://www.nuget.org/packages/MSFT.ParallelExtensionsExtras/ – Reyhn Aug 03 '16 at 09:08
  • 1
    A more recent article on this combination by Can Bilgin - [link](https://canbilgin.wordpress.com/2017/02/05/curious-case-of-parallel-foreach-with-blockingcollection/) – MuKa May 16 '18 at 13:50
  • link to blog: [ParallelExtensionsExtras Tour – #4 – BlockingCollectionExtensions](https://devblogs.microsoft.com/pfxteam/parallelextensionsextras-tour-4-blockingcollectionextensions/) – JonasH Mar 03 '21 at 16:02