3

I have a program written in C that allows the user to scroll through a display of about a zillion small files. Each file needs to undergo a certain amount of processing (read only) before it's displayed to the user. I've implemented a buffer that preprocesses the files in a certain radius around the user's position, so if they're working linearly through them, there's not much delay. For various reasons, I can only actually run my processing algorithm on one file at a time (though I can have multiple files open, and I can read from them) so my buffer loads sequentially. My processing algorithms are as optimized as they're going to get, but I'm running into I/O problems. At first, my loading process is slow, but when the files have been accessed a few times, it speeds up by about 5x. Therefore I strongly suspect that what's slowing me down is waiting for the Windows page cache to pull my files into memory. I know very little about that sort of thing. If I could ensure my files were in the cache before my processing algorithm needed them, I'd be in business.

My question is: is there a way to persuade/cajole/trick/intimidate Windows into loading my files into the page cache before I actually get around to reading/processing them?

jsn
  • 163
  • 2
  • 9

2 Answers2

1

There's only one way to get a file into the file system cache: reading it. That's a chicken-and-egg problem. You can get the egg first by using a helper thread that reads files. It would have to have some kind of smarts about what file is likely to be next. And not read too much.

Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • Probably it's enough to read just one byte per page (4KB?) to have the complete page read and cached. – Wim Jan 18 '10 at 22:40
  • If the method listed in the other comment doesn't improve performance, I'm going to try reading the minimal amount per block, thanks. – jsn Jan 29 '10 at 16:08
1

On a POSIX system, you'd use posix_fadvise:

POSIX_FADV_WILLNEED

        Specifies that the application expects to access the specified data in the near future.

However, that doesn't seem to exist on Windows. What is fadvise/madvise equivalent on windows ? - Stack Overflow has some alternatives.

Community
  • 1
  • 1
ephemient
  • 198,619
  • 38
  • 280
  • 391
  • Thanks, this is helpful. I will update if I end up using this and notice it actually improves performance. – jsn Jan 29 '10 at 16:07