Is there a quicker way of updating the UI when iterating through a large list?

Question

I did three tests on a Dictionary that has about 350 thousand items. I needed to see how many items I could iterate over a 30 second period.

My initial test was just loop through the dictionary with no checks and no UI/console updating. It completed just a second after starting the test.

The second test was writing the count to the console. About 23,600 items were reached.

foreach (KeyValuePair<UInt64, MftFile> entry in mftFiles)
{
    fileCount++;
    Console.WriteLine(fileCount.ToString());
}

I then tested how quickly it would run when updating the form with the count. It reached just a little over 16000 items.

foreach (KeyValuePair<UInt64, MftFile> entry in mftFiles)
{
    fileCount++;
    MessageReceived(this, GetMessageReceivedEventArgs("Proactive Checks - RIFT",
    string.Format("Analyzing Drive {0} - MFT Record {1} of {2}", drive.Name, fileCount.ToString(), mftFiles.Count.ToString()), string.Empty));
}

Doing any kind of conditional logic in the loop slows the loop down by a huge amount. I originally created a stop watch outside of the loop and checked it inside of the loop to see when it had reached a certain time. Over a minute went by and it had only iterated over ~5000 items.

There's a C++ app that does a similar thing to what I'm doing. It takes less than 60 seconds for it to iterate through all 300,000 items, update the UI, find all duplicates, and create hashes for each duplicate.

Is there a better way to iterate through the dictionary?

Answer 1

You are already using quicker way to iterate through the dictionary. There is no other quicker way to iterate through the dictionary. See What is the best way to iterate over a Dictionary in C#?

If you want to stop after reaching certain time then this might work

var t = new Thread(() => {
   foreach (KeyValuePair<UInt64, MftFile> entry in mftFiles)
   {
       fileCount++;
       Console.WriteLine(fileCount.ToString());
   }
});
t.Start();
// Sleep the current thread as long as you want to run the task
Thread.Sleep(/* Specify time in milliseconds */);
// After that abort the Thread to exit the job
t.Abort();

Further use StringBuilder instead of String.Format as it has to parse the string that takes time.

EDIT To quickly update the UI Run MFT Scan in other thread and Watch Scan in Second thread. Instead of updating UI on each file, update after specific time. I achieved a good performance on updating UI by using below code.

public MyForm() {
    InitializeComponent();

    scanProgress = new Action(() => {
        MessageReceived(this, GetMessageReceivedEventArgs("Proactive Checks - RIFT", string.Format("Analyzing Drive {0} - MFT Record {1} of {2}", drive.Name, cMft, count, string.Empty));
    });
}

Action<> scanProgress;

int cMft = 0;
int count;
void ScanMft() {
    count = mftFiles.Count;
    foreach (KeyValuePair<UInt64, MftFile> entry in mftFiles) {
        cMft++;
        /* Scan MFT */
    }
}

void WatchScan() {
    while (cMft < count) {
       Thread.Sleep(200); 
       this.BeginInvoke(scanProgress);
    }
}

void RunScan() {
    new Thread(ScanMft).Start();
    new Thread(WatchScan).Start();
}

Answer 2

There are 2 questions there, how to do it faster (you're not providing enough code for that, your bottleneck definately shouldn't be iterating a collection by itself, and those numbers you're getting are painfully slow, are you testing in debug? NEVER EVER PERFORMANCE TEST IN DEBUG!)

Post more code and test it in release, do not post sample tests but your actual code and we can help on that.

The second part of the question is, how to avoid blocking the UI while you're processing, it's simple, don't process on the main thread, move your whole function to another thread and periodically (every 100 ish? inserts) update the control by using Control.BeginInvoke http://msdn.microsoft.com/en-us/library/system.windows.forms.control.begininvoke(v=vs.110).aspx

This will mean the UI is fully reactive during your processing, and only frezzes for tiny time slices while it's actually being updated so it would be responsive all of the time.

Another optimization if all of this doesn't suffice would be, don't do work you don't need to do! Stick to what i said (processing on another thread) but don't update the UI except for the 1st 200 or 300 elements, then implement a virtualizing control that only displays what should currently be seen (trivial to do in WPF but i'm assuming you have to roll your own in winform or buy a component). Note that if you're early in the project i would strongly suggest you switch to WPF as your application sounds like it could use it (not a trivial business apps with a couple buttons / labels)

Edit : just to be clear, enumerating a dictionary could not possibly be your bottleneck, i just tested on my mid range machine and i enumerate a million keyvaluepairs in a dictionary in . . 12 milliseconds!

Edit 2 : however Console.WriteLine definately could be your bottleneck, which is why you shouldn't introduce unrelated APIs when testing, iterating to the next item = no work, building up the string = a little work, actually passing it to the console for printing = lots of work

Answer 3

You have already answered your own question. You have proven that you can iterate through the dictionary in under a second, so it doesn't sound like you need to increase the speed of that- the slowness has nothing to do with dictionary. It's the MessageReceived function call. I don't know what that function does so I cannot troubleshoot why it is slow.

I also would try removing this:

mftFiles.Count.ToString()

You can do that once outside of the loop and store the value, which might help a little.

Answer 4

You can create a worker thread and put foreach loop in it. By doing this both UI and foreach loop will run on different thread, so you dont need to update the UI.

Another alternate way (which I don't recommend because it has its own consequences) is to put

Application.DoEvent();

in foreach loop. Doing so, your UI will get updated with each iteration.