119

What tools or techniques can I use to remove cached file contents to prevent my performance results from being skewed? I believe I need to either completely clear, or selectively remove cached information about file and directory contents.

The application that I'm developing is a specialised compression utility, and is expected to do a lot of work reading and writing files that the operating system hasn't touched recently, and whose disk blocks are unlikely to be cached.

I wish to remove the variability I see in IO time when I repeat the task of profiling different strategies for doing the file processing work.

I'm primarily interested in solutions for Windows XP, as that is my main development machine, but I can also test using linux, and so am interested in answers for that environment too.

I tried SysInternals CacheSet, but clicking "Clear" doesn't result in a measurable increase (restoration to timing after a cold-boot) in the time to re-read files I've just read a few times.

tvanfosson
  • 524,688
  • 99
  • 697
  • 795
Stephen Denne
  • 36,219
  • 10
  • 45
  • 60
  • possible duplicate of [How to invalidate the file system cache?](http://stackoverflow.com/questions/7405868/how-to-invalidate-the-file-system-cache) – user541686 Dec 13 '12 at 21:18
  • 2
    Why was this question [deleted](http://meta.stackexchange.com/questions/186129/please-undelete-this-moderator-deleted-question-on-so)? – Dan Dascalescu Aug 12 '15 at 22:02

9 Answers9

97

Use SysInternal's RAMMap app.

rammap empty standby

The Empty / Empty Standby List menu option will clear the Windows file cache.

Jeff Atwood
  • 63,320
  • 48
  • 150
  • 153
smallest
  • 986
  • 7
  • 2
  • 2
    RAMMap doesn't run on Windows XP. Do those menu options do something different from the "Clear" button in SysInternals Cacheset application? – Stephen Denne Mar 12 '12 at 21:32
  • @stephen this technique worked perfectly for me under Windows 8! Windows XP is 12 years old now, any reason you can't try this on Vista, Windows 7, or Windows 8? – Jeff Atwood Nov 13 '12 at 05:06
  • @Jeff I no longer work with those limitations. So "Empty Standby List" results in the next read from a recently read file requiring disk IO? – Stephen Denne Nov 13 '12 at 22:43
  • 2
    Press F5 to refresh the file list after clicking empty standby list – JoshG Aug 20 '13 at 00:31
  • `The Empty / Empty Standby List menu option will clear the Windows file cache.` How do you know that? How is one to make the connection between that menu item and the disk-cache? – Synetech Dec 01 '13 at 00:49
  • @smallest - thanks, this was invaluable in testing disk performance for some algorithm optimisation we were attempting! – Jon Cage Jan 29 '14 at 16:31
  • 3
    For a programmatic approach, see this SO post: http://stackoverflow.com/a/23085045/430360 – snemarch Nov 11 '15 at 08:35
17

For Windows XP, you should be able to clear the cache for a specific file by opening the file using CreateFile with the FILE_FLAG_NO_BUFFERING options and then closing the handle. This isn't documented, and I don't know if it works on later versions of Windows, but I used this long ago when writing test code to compare file compression libraries. I don't recall if read or write access affected this trick.

  • Works perfectly for me under Windows 7 SP1, x64. Great tip! – cxxl Oct 22 '12 at 16:26
  • 2
    So that must be repeated for *every file*? So if for example, you copy a directory containing 100MB spread across 30 files in 10 subdirectories, you must open each and every one separately to make sure you read the actual disk instead of the cache? – Synetech Dec 01 '13 at 00:51
  • Other answers on the site affirm this works in Win7 and 8 as well. I presume it works on vista. And yes, you have to run it on each file, but it doesn't take all that long. All you need to do is open and close each file, and when you close, Windows clears the cache. After you've done this, THEN you run your performance test. – Mooing Duck Jan 04 '14 at 00:36
  • 1
    This was a perfect answer. Simple to code, effective, and should work on most windows OSes (including XP, I've only tested on Win7x64). I opened the files with read privileges and no sharing. Not sure what combo matters. – Rosco Aug 06 '14 at 17:06
  • Similar functionality on Linux (2.4.10 onward) with the O_DIRECT flag in the open(2) syscall. Well, more precisely, from what I understand, O_DIRECT does not clear the cache for that file, it tries it's best to bypass it. – kaiwan Jul 15 '15 at 12:06
  • This is THE solution that worked for me. Call CreateFile with FILE_FLAG_NO_BUFFERING as the second to last parameter, then immediately close the file. Afterwards open using regular C++ API ifstream, which is now considerably slower due to no caching. – Markus Apr 30 '19 at 11:29
15

A command line utility can be found here

from source:

EmptyStandbyList.exe is a command line tool for Windows (Vista and above) that can empty:

  • process working sets,
  • the modified page list,
  • the standby lists (priorities 0 to 7), or
  • the priority 0 standby list only.

Usage:

EmptyStandbyList.exe workingsets|modifiedpagelist|standbylist|priority0standbylist
Edwin van Mierlo
  • 2,398
  • 1
  • 10
  • 19
  • 4
    Highly underrated answer amidst all the noise, simple utility that does one job and does it well. You should probably include more details in the answer to get the upvotes you deserve. – Hashim Aziz Sep 30 '18 at 22:50
  • 1
    @HashimAziz, I concur! It is worth a short explanation of what those four things are that it can empty. I for example just want to get rid of the disk read cache. I'm testing file load times and get huge performance increases if I recently loaded that same file which we have concluded is probably because of a disk read cache in Windows (and how I land here). This utility looks grand but now I have to go research what these 4 stanby lists are and which one I want to clear. – Bernd Wechner Aug 02 '21 at 01:22
  • 1
    I'm using this to great effect in my tests now. Thanks. It's not well documented. I infer empirically that without arguments it just empties all the four mentioned things or at least the 8 standby lists. Either way, I run it with not arguments between test runs and I have seen a cycle of slowload, fastload, fastload change to slowload, slowload, slowload. Much as I was expecting and am happy. I was witnessing enormous file load speed ups on subsequent iterations of this test without explanation, well until we found this was the likely cause, and tested this utility and can confirm it works! – Bernd Wechner Aug 03 '21 at 03:18
  • 1
    Further I run it form Python test scripts. Which throws up a console while it runs. Would love to find a native Python solution, but haven't found any bindings yet. – Bernd Wechner Aug 03 '21 at 03:19
14

A quick googling gives these options for Linux

  1. Unmount and mount the partition holding the files
  2. sync && echo 1 > /proc/sys/vm/drop_caches
0xC0000022L
  • 20,597
  • 9
  • 86
  • 152
John Nilsson
  • 17,001
  • 8
  • 32
  • 42
  • 2
    Thanks, that looks very useful, though I'd possibly want to echo 3 instead of 1. I'm primarily interested in Windows XP, which is why I hadn't found that in my googling. – Stephen Denne Jan 25 '09 at 22:03
  • Unfortunately the linux environment I could make use of this in, has kernel version 2.6.9. drop_caches was added in kernel 2.6.16 – Stephen Denne Jan 25 '09 at 22:45
  • Lots of other posts I have read suggest that you can unmount and mount the filesystem to remove all the chached items for it, I would assume that was the case before 2.6.16 as well as in more recent kernels. – TafT Aug 19 '11 at 14:59
4
 #include <fcntl.h>

int posix_fadvise(int fd, off_t offset, off_t len, int advice);

with advice option POSIX_FADV_DONTNEED:
The specified data will not be accessed in the near future.

Prasad Jadhav
  • 5,090
  • 16
  • 62
  • 80
Mark
  • 41
  • 1
  • Only for Linux and other posix, not for Windows: http://stackoverflow.com/questions/29752064/what-is-posix-fadviseposix-fadv-dontneed-equivalent-on-windows http://stackoverflow.com/questions/1201168/what-is-fadvise-madvise-equivalent-on-windows – osgx Apr 18 '17 at 00:47
3

For a much better view of the Windows XP Filesystem Cache - try ATM by Tim Murgent - it allows you to see both the filesystem cache Working Set size and Standby List size in a more detailed and accurate view. For Windows XP - you need the old version 1 of ATM which is available for download here since V2 and V3 require Server 2003,Vista, or higher.

You will observe that although Sysinternals Cacheset will reduce the "Cache WS Min" - the actual data still continues to exist in the form of Standby lists from where it can be used until it has been replaced with something else. To then replace it with something else use a tool such as MemAlloc or flushmem by Chad Austin or Consume.exe from the Windows Server 2003 Resource Kit Tools.

robertcollier4
  • 742
  • 10
  • 10
3

I've found one technique (other than rebooting) that seems to work:

  1. Run a few copies of MemAlloc
  2. With each one, allocate large chunks of memory a few times
  3. Use Process Explorer to observe the System Cache size reducing to very low levels
  4. Quit the MemAlloc programs

It isn't selective though. Ideally I'd like to be able to clear the specific portions of memory being used for caching the disk blocks of files that I want to no longer be cached.

Stephen Denne
  • 36,219
  • 10
  • 45
  • 60
  • That has the side-effect of pushing stuff into the pagefile which will kill performance for quite a while afterwards. If you are going to resort to a kludge, then you may as well just read some other large file; at least that only clears the disk cache and nothing else. – Synetech Dec 01 '13 at 00:52
0

As the question also asked for Linux, there is a related answer here.

The command line tool vmtouch allows for adding and removing files and directories from the system file cache, amongst other things.

seeker
  • 671
  • 7
  • 13
0

There's a windows API call https://learn.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-setsystemfilecachesize that can be used to flush the file system cache. It can also be used to limit the cache size to a very small value. Looks perfect for these kinds of tests.

Geoff Johnson
  • 47
  • 2