adding data rows using parallel.For

Question

I have n sets of rows in a data table and I want to add these n sets. I am looking at ways to parallelize this.

example: Set 1: row 0 to row 9 Set 2: row 10 to row 19 Set 3: row 20 t0 row 29 Set 3: row 20 t0 row 29

Add these to another table Total datatable (row 0 to row 9) = set 1 + set 2 + set 3

I am using C#.

Currently I have implemented this using ThreadPool.QueueUserWorkItem as follows but it is bit ugly.

I span n number of threads (max = total cpus) distribute sets to these threads and calculate partial sums span n/2 threads to sum these partial sums when I have two partial sums, then I sum them to total dataset.

example: if I have 10 sets and 4 cpus

on first iteration

1st cpu: sum 0,1,2 = sum1 2nd cpu: sum 3,5 = sum2 3rd cpu: sum 6,7 = sum3 4th cpu: sum 8,9 = sum4

second iteration

1st thread sum1 + sum2 = sum10 2nd thread sum3 + sum4 = sum 11

third and final iteration

total sum = sum10 + sum11

I am trying to do this using .net framework 4.0 Parallel library.

Answer 1

The Microsoft Patterns and Practices group has a book which details a number of techniques for parallel programming. There is a whole chapter on Parallel Aggregation which can be found at http://msdn.microsoft.com/en-us/library/ff963547.aspx and this contains code examples in C#.

Answer 2

Please see:

Parallel ForEach on DataTable

From MSDN:

This type is safe for multithreaded read operations. You must synchronize any write operations.

http://msdn.microsoft.com/en-us/library/system.data.datatable.aspx