How to flush the CPU cache in Linux from a C program?

Question

I am writing a C program in which I need to flush my memory. I would like know if there is any UNIX system command to flush the CPU cache.

This is a requirement for my project which involves calculating the time taken for my logic.

I have read about the cacheflush(char *s, int a, int b) function but I am not sure as to whether it will be suitable and what to pass in the parameters.

Answer 1

1. I take it you mean "CPU cache", not memory cache

2. The link above is good: the suggestion "write a lot of data via CPU" is not Windows specific

3. Here's another variation on the same theme:

   • How to clear CPU L1 and L2 cache

4. Here's an article about Linux and CPU cache:

   • http://lwn.net/Articles/252125/

NOTE:

At this (very, very low) level, "Linux" != "Unix"

Answer 2

This is how Intel suggests flushing the cache:

mem_flush(const void *p, unsigned int allocation_size){
    const size_t cache_line = 64;
    const char *cp = (const char *)p;
    size_t i = 0;

    if (p == NULL || allocation_size <= 0)
            return;

    for (i = 0; i < allocation_size; i += cache_line) {
            asm volatile("clflush (%0)\n\t"
                         : 
                         : "r"(&cp[i])
                         : "memory");
    }

    asm volatile("sfence\n\t"
                 :
                 :
                 : "memory");
}

Answer 3

If you're writing a user-mode (not kernel-mode) program, and if it's single-threaded, then there's really no reason for you to ever bother flushing your cache in the first place. Your user-mode program can just forget that it even exists; it's just there to speed up your program's execution, and the OS manages it via the processor's MMU.

There are only a couple reasons I can think of that you might actually want to flush the cache from your user-mode application:

1. Your app is intended to run on a symmetric multiprocessor system, or has data transactions with external hardware)
2. You're simply testing your cache for some sort of performance test (in which case you should probably really should be writing your test to operate in kernel mode, perhaps as a driver).

In any case, assuming you're using Linux...

#include <asm/cachectl.h>

int cacheflush(char *addr, int nbytes, int cache);

This assumes you have a block of memory you just wrote to and you want to make sure it's flushed out of the cache back to main memory. The block begins at addr, and it's nbytes long, and it's in one of the two caches (or both):

   ICACHE Flush the instruction cache.
   DCACHE Write back to memory and invalidate the affected valid cache lines.
   BCACHE Same as (ICACHE|DCACHE).

Normally you'd only need to flush the DCACHE, since when you write data to "memory" (i.e. to the cache), it's normally data, not instructions.

If you want to flush "all of the cache" for some strange testing reason, you could malloc() a big block that you know is larger than your CPU's cache (shoot, make it 8 times as big!), write any old garbage into it, and just flush that entire block.

See also: How to perform cache operations in C++?

Answer 4

OK, sorry about my first answer. I later read your follow-up comments below your question, so I realize now that you want to flush the INSTRUCTION CACHE to boot your program (or parts of it) out of the cache, so that when you test its performance, you also test its initial load time out of main memory into the instruction cache. Do you also need to flush any data your code will use out to main memory, so that both data and code are fresh loads?

Before anything else, I'd like to mention that main memory itself is also a form of cache, with your hard disk (either the program on disk, or swap space on disk) being the lowest, slowest place your program's instructions could be coming from. That said, when you first run through a routine for the first time, if it hasn't already been loaded into main memory from disk by virtue of being near other code that has already executed, then its CPU instructions will first have to be loaded from disk. That takes an order of magnitude or more longer than loading it from main memory into the cache. Then once it's loaded into main memory, it takes somewhere along the lines of an order of magnitude longer to load from main memory into the cache than it takes to load from the cache into the CPU's instruction fetcher. So if you want to test your code's cold-start performance, you have to decide what cold-start means.... pulling it out of disk, or pulling it out of main memory. I don't know of any command to "flush" instructions/data out of main memory out to swap space, so flushing it out to main memory is about as much as you can do (that I know of), but keep in mind that your test results may still differ from the first run (when it may be pulling it off disk) to subsequent runs, even if you do flush the instruction cache.

Now, how would one go about flushing the instruction cache to ensure that their own code is flushed out to main memory?

If I needed to do this (very odd thing to do in my opinion), I'd probably start by finding the length & approximate placement of my functions in memory. Since I'm using Linux, I'd issue the command "objdump -d {myprogram} > myprogram.dump.txt", then I'd open myprogram.dump.txt in an editor and search for the functions I want to flush out, and figure out how long they are by subtracting their end address form their start address using a hex calculator. I'd write down the sizes of each. Later I'd add cacheflush() calls in my code, giving it the address of each function I want to flush out as 'addr' and the length I found as 'nbytes', and ICACHE. Just for safety I'd probably fudge a little & add about 10% to the size, just in case I make a few tweaks to the code and forget to adjust the nbytes. I'd make a call to cacheflush() like this for each function I want to flush out. Then if I need to flush out the data also, if it's using global/static data, I can flush those also (DCACHE), but if it's stack or heap data, there's really nothing realistic that I can (or should) do to flush that out of cache. Trying to do so would be an exercise in silliness, because it would be creating a condition that would never or very rarely exist in normal execution. Assuming you're using Linux...

#include <asm/cachectl.h>

int cacheflush(char *addr, int nbytes, int cache);

...where cache is one of:
   ICACHE Flush the instruction cache.
   DCACHE Write back to memory and invalidate the affected valid cache lines.
   BCACHE Same as (ICACHE|DCACHE).

BTW, is this homework for a class?