3

Before refactoring my code to start experimenting, I'm hoping the wisdom of the community can advise me on the correct path.

Problem: I have a WPF program that does a diff on hundreds of ini files. For performance of the diffing I'd like to keep several hundred of the base files that other files are diffed against in memory. I've found using custom classes to store this data starts to bring my GUI to a halt once I've loaded 10-15 files with approximately 4000 line of data each.

I'm considering several strategies to improve performance:

  • Don't store more than a few files in memory at a time and forget about what I hoped would have been perf improvement in parsing by keeping them in memory
  • Experiment with running all the base file data in a BackgroundWorker thread. I'm not doing any work of these files on the GUI thread but maybe all that stored data is affecting it somehow. I'm guessing here.
  • Experiment with System.Runtime.Caching class.

The question asked here on SO didn't, in my mind, answer the question of what's the best strategy for this type of work. Thanks in advance for any help you can provide!

Community
  • 1
  • 1
Roger
  • 107
  • 6

3 Answers3

3

Assuming 100 character lines of text 15 * 4000 * 100 is only 6MB which is a trivial amount of memory on a modern PC. If your GUI is coming to a halt then to me that is an indication of virtual memory being swapped in and out to disk. That doesn't make sense for only 6MB so I'd figure out how much it's really taking up and why. If may well be some trivial mistake that would be easier to fix than re-thinking your whole strategy. The other possibility is that it has nothing to do with memory consumption but rather an algorithm issue.

Mike W
  • 1,276
  • 8
  • 10
  • in .net, it is using unicode, I think char = 2bytes, so the memory consumption at least double the size of your calculation, and also the size of memory consumption tightly depends on the data type. – Johnny Jan 15 '12 at 05:25
  • @Johnny: Good point. I was thinking in terms of the size of traditional ansi-only text files on disk. Even 12MB is trivial so until we hear back from the OP we will never know. – Mike W Jan 15 '12 at 19:34
  • I changed the strategy I was employing for using a RichTextBox to get the performance I was seeking, essentially only showing the data that could possibly fit in the visible screen rather than having an entire file color formatted within the RTB. It's made for an interesting learning experience because RTB is a fairly terrible container if you want to do any type of large formatted documents. – Roger Jan 25 '12 at 00:22
3

You should use MemoryCache for this.

It works almost alike the ASP.Net Cache class, and allows you to set when it should clean up, which should be cleaned up first etc.

It also allows you to reload items based on dependencies, or after a certain time. Has callbacks on remove.

Very complete.

Aidiakapi
  • 6,034
  • 4
  • 33
  • 62
  • Sorry for my very late response. But yes, it does. Just as the link I gave you clearly states: 'The main differences between the Cache and MemoryCache classes are that the MemoryCache class has been changed to make it usable by .NET Framework applications that are not ASP.NET applications' – Aidiakapi Jan 16 '12 at 00:10
0

if your application start to hang, it more like you are doing intensive process in your GUI process, which consumer too much resource either CPU/Memory on your GUI thread, thus the GUI thread can't repaint your UI in time.

The best way to resolve it is spawn separate thread to do the diff operation, as you mentioned in your post, you can use backgroundworker, or you can use threadpool to spawn as much thread as you can to do the diff.

Don't think you need to cache the file in memory, I think it would be more appropriate to save the result into file, and load the file ondemand. it shouldn't become a bottleneck of your application.

Johnny
  • 363
  • 1
  • 8