Is PLinq Inherently Faster than System.Threading.Tasks.Parallel.ForEach

Question

Summary: I changed from System.Threading.Tasks.Parallel.ForEach and Concurrent Data structure to a simple plinq (Parallel Linq) query. The speed up was amazing.

So is plinq inherently faster than Parallel.ForEach? Or is it specific to the task.

// Original Code
// concurrent dictionary to store results
var resultDict = new ConcurrentDictionary<string, MyResultType>();

Parallel.ForEach(items, item =>
        {
            resultDict.TryAdd(item.Name, PerformWork(source));
        });


// new code

var results =
            items
            .AsParallel()
            .Select(item => new { item.Name, queryResult = PerformWork(item) })
            .ToDictionary(kv => kv.SourceName, kv => kv.queryResult);

Notes: Each task (PerformWork) now runs between 0 and 200 ms. It used to take longer before I optimized it. That's why I was using the Tasks.Parallel library in the fist place. So I went from 2 seconds total time to ~100-200 ms total time, performing roughly the same work, just with different methods. (Wow linq and plinq are awesome!)

Questions:

Is the speed up due to using plinq vs Parallel.ForEach?
Is it instead simply the removal of the concurrent data structure (ConcurrentDictionary)? (Because it doesn't need to synchronize threads).
Based on the answer from this related question

Whereas PLINQ is largely based on a functional style of programming with no side-effects, side-effects are precisely what the TPL is for. If you want to actually do work in parallel as opposed to just searching/selecting things in parallel, you use the TPL.

Can I assume that because my pattern is basically functional (giving inputs produce new outputs without mutation), that plinq is the correct technology to use?

I'm looking for validation that my assumptions are correct, or an indication that I'm missing something.

What data type is your "items" variable and how many items are in it? — Drew Marsh, Mar 04 '11 at 22:13

score 4 · Answer 1 · answered Mar 04 '11 at 16:12

4

It's not possible to use these 2 code samples to do a definitive comparison between Parallel.ForEach and PLINQ. The code samples are simply too different.

The first item that jumps out at me is the first sample uses ConcurrentDictionary and the second uses Dictionary. These two types have very different uses and performance characteristics. In order to get an accurate comparison between the two technologies you need to be consistent here with the types.

answered Mar 04 '11 at 16:12

JaredPar

733,204
149
1,241
1,454

How would it be possible to load a regular generic dictionary with Parallel.ForEach? – Chris Weber Mar 04 '11 at 16:42
@Chris i'm not sure it's possible without some sort of locking. – JaredPar Mar 04 '11 at 16:49
@JarePar right. That's the context of my question (though probably not stated well). How do I do something in parallel and return a dictionary. In this case, plinq was way faster. Is this generalizable to a class of problems or simply a quirk with ConcurrentDictionary. – Chris Weber Mar 04 '11 at 17:06
@Chris, it looks like your primary objective here is to do a projection operation in parallel. If so then I think PLINQ is better here, not necessarily for performance reasons but because it presents an API which directly supports the scenario. `Parallel.Foreach` doesn't provide the same projection capabilities AFAIK. – JaredPar Mar 04 '11 at 17:16
Actually, the fact that the PLINQ example uses ToDictionary should theoretically make it slower since items can not be added in parallel unlike the ConcurrentDictionary. In fact, if you look at ParallelEnumerable's implementation of ToDictionary it actually pulls items from the upstream parallel query as they become available in serial fashion to add them to the Dictionary one at a time. So, in that regard at least, his Parallel::ForEach should perform better. See my answer for why I think it's not. – Drew Marsh Mar 04 '11 at 22:37

Drew Marsh · Accepted Answer · 2011-03-04T22:40:22.057

Based on the limited information you've provided in your sample (I asked for more details in a comment on the OP), I'm guessing sure you're seeing differences due to the partitioning algorithm that is used. You should read up on Chunk Partitioning vs. Range Partitioning in this blog post where he discusses how they differ and for which types of work they might be best suited for. Highly recommend you read that blog article as well as this one which goes into a little more detail on those two types along with two other types of partitioning that can be used, though not applicable to your sample, as well as giving some visual aids to better understand the partitioning. Finally, here's yet another blog post that discusses work partitioning and how it can affect you when the default partitioning algorithm doesn't make sense for your particular workload. That post actually refers to a great program that helps you visualize the partitioners at work that's part of a set of parallel samples from the PFX team.

Is PLinq Inherently Faster than System.Threading.Tasks.Parallel.ForEach

2 Answers2

Linked