-2

I have IEnumerable collection as shown below. this is collection of collection. Main collection has 10 items and each item has 100k items.

My CPU has 6 cores, 24GB RAM, 1.5TB hard disk. 3 user does the remote desktop.

 IEnumerable<AFValues> afValues = listAvgResults.Select(pointResults => pointResults[type]);

when i use Parallel.ForEach it took 9.8 and foreach took 10 sec.

MTObservableCollection<VesselData> vesselList = new MTObservableCollection<VesselData>();

Parallel.ForEach(afValues, pointValues =>
                        {
 vesselList.Add(row);
});

Thanks

Prasad
  • 21
  • 3
  • 1
    why would you expect a simple add to be sped up? Also, where's your lock? – Mitch Wheat Dec 14 '18 at 10:45
  • First determine where your bottleneck is and whether it's CPU-bound or I/O-bound. `Parallel.ForEach` can speed up CPU-bound work but only if it's appropriate to parallelize the activity (e.g.if you need locks around the actual CPU consuming activity it won't help). Profile first, then attempt to optimize. – Damien_The_Unbeliever Dec 14 '18 at 10:49
  • Whether it is fast or not is irrelevant, if it isn't **safe**. `ObservableCollection` is not thread-safe. You can't `Add` to it safely from multiple threads. Or are you using something like https://stackoverflow.com/a/23108315/34092 (in which case it is thread-safe, but only because it is effectively **single threaded**)? – mjwills Dec 14 '18 at 11:00
  • What are you using these large volumes of items for? For displaying to the user? Something else? – mjwills Dec 14 '18 at 11:07
  • @mjwills displaying to the user. User can filter , report plot etc – Prasad Dec 14 '18 at 12:23
  • @MitchWheat i guess you dint read full question. Inside each foreach there is some logic (very primitive like comparison) happening – Prasad Dec 14 '18 at 12:26
  • 1
    `i guess you dint read full question. Inside each foreach there is some logic (very primitive like comparison) happening` I have re-read the question now multiple times and can't find the text that states that. Can you point it out to us? Perhaps change the text colour / boldness in your question to highlight it better? – mjwills Dec 14 '18 at 21:33
  • @Prasad: I read the full question several times. The text " Inside each foreach there is some logic (very primitive like comparison)" does not appear. – Mitch Wheat Dec 14 '18 at 23:06

3 Answers3

1

The bottleneck will be probably adding to the vesselList where the adding will be probably blocking operation. So other threads probably waits for other threads alreading adding rows into list.

If you want precise results, use some of profiler.

TcKs
  • 25,849
  • 11
  • 66
  • 104
1

All the work happens in Add(), and assuming MTObservableCollection uses some kind of locking in that method, all threads are waiting on each other to add something to the list.

This does not sound like work that can be sped up by parallelization.

CodeCaster
  • 147,647
  • 23
  • 218
  • 272
-2

If you are using MSSQL, an alternative would be to do BULK insert.

There are 2 ways that I've done that in the past:

  1. SqlBulkCopy

  2. Stored Procedure with Table-Valued Parameters (and send the data as DataTable from C#)

This way it'll be less then a second i think.

Stefan Taseski
  • 242
  • 2
  • 24