2

I am searching for a way to read text file content very fast, for example: 2GB file in under 20 seconds (without loading the entire file into memory, thus keeping the memory usage low).

I have read here about Memory Mapped Files. The CreateFromFile method can create a file mapping in memory. I have tested this code:

using(var mmf1 = MemoryMappedFile.CreateFromFile(file, FileMode.Open)) {
    using(var reader = mmf1.CreateViewAccessor()) {
        //...
    }
}

My question is: is this the correct way to index the file into memory so that it will be available for fast search?

Once the file has been mapped/indexed into memory, I hope to use a Parallel loop to search the file for content. Right now I am testing this implementation. If I understand it correctly, it uses 1 thread to read a line in a file, and the 2nd thread to check/process the line.

Is it possible to use 1 thread to read and process the file content from top to bottom and the 2nd thread to read file content from the bottom up, thus increasing the search speed by a factor of 2.

Because as soon as the needed content has been found, all threads must stop search.

Mcd
  • 21
  • 4
  • 1
    Indeed it is possible. Whether it is good idea or not - you need to try it out and measure yourself. There is very good chance that trying to read *large* amount of data in parallel will be slower than sequential scan. – Alexei Levenkov Jan 29 '16 at 00:10
  • My understanding is such, that MMFiles optimise chunked access to a file for multiple processees (http://stackoverflow.com/a/1859843/444149) . But if You need to search a large text - I think first thing is to create an index.. Not sure what You're searching for.. but maybe building a Index of words would help ? – Marty Jan 29 '16 at 00:21
  • You want the fastest disc read possible and that is sequential read from the beginning to the end. You are limited by hardware, memory mapped files would not help You with this. – Antonín Lejsek Jan 29 '16 at 03:16
  • I am searching for a way to populate a listView/listBox very fast, just like Large Text File Viewer for example. I am reading everywhere that it is not recommended to load the entire file into memory. Large Text File Viewer loads a file of 1GB into a listbox/listview within 10 seconds and the memory never exceeds 10 MB! How is that done? – Mcd Feb 02 '16 at 00:35

0 Answers0