What is cache in C++ programming?

Question

Firstly I would like to tell that I come from a non-Computer Science background & I have been learning the C++ language. I am unable to understand what exactly is a cache? It has different meaning in different contexts. I would like to know what would be called as a cache in a C++ program? For example, if I have some int data in a file. If I read it & store in an int array, then would this mean that I have 'cached' the data? To me this seems like common sense to use the data since reading from a file is always bad than reading from RAM. But I am a little confused due to this article.

In a CPU there can be several caches, to speed up instructions in loops or to store often accessed data. These caches are small but very fast. Reading data from cache memory is much faster than reading it from RAM.

It says that reading data from cache is much faster than from RAM. I thought RAM & cache were the same. Can somebody please clear my confusion?

EDIT: I am updating the question because previously it was too broad. My confusion started with this answer. He says

RowData and m_data are specific to my implementation, but they are simply used to cache information about a row in the file

What does cache in this context mean?

"Cache" means lots of different things in computing. It would be a waste of time explaining them when wikipedia already does so: http://en.wikipedia.org/wiki/Cache — Jonathan Wakely, Sep 18 '13 at 17:42
In general, a cache keeps a copy of data that's slow to access somewhere that's faster to access. So a network location might be cached on a local disk, a disk file might be cached in RAM, and RAM might be cached in faster on-chip memory. — Mike Seymour, Sep 18 '13 at 17:45
The article is referring to hardware caches. Basically, there are different kinds of RAM. The faster ones are more expensive, and hold less memory. A good operating system will try to predict what to put in this faster cache and what to leave in regular RAM to make things go as fast as possible, but it's transparent to most if not all application programmers. — Scott Mermelstein, Sep 18 '13 at 17:45
Cache is a more general concept, the word is used in many contexts. That definition mentions several different types - a CPU cache, disk cache, browser cache. So a "cache" is not just one specific thing, it's used to mean the part of the system being described that holds data for faster retrieval than the original source. If you quote the original text that uses the word, someone could help you understand what it means in that context. — Brad Peabody, Sep 18 '13 at 17:46
See: [What is “cache-friendly” code?](http://stackoverflow.com/q/16699247/1168156) — LihO, Sep 18 '13 at 17:47
I have updated the question to specify the context of the usage. — Cool_Coder, Sep 18 '13 at 17:58
@Cool_Coder You should also quote the article you linked to - at some point, the link or its data may change, and for this to be useful to future readers, the text should be copied directly into your question. — Scott Mermelstein, Sep 18 '13 at 18:07

score 22 · Accepted Answer · answered Sep 18 '13 at 18:21

Any modern CPU has several layers of cache that are typically named things like L1, L2, L3 or even L4. This is called a multi-level cache. The lower the number, the faster the cache will be.

It's important to remember that the CPU runs at speeds that are significantly faster than the memory subsystem. It takes the CPU a tiny eternity to wait for something to be fetched from system memory, many, many clock-cycles elapse from the time the request is made to when the data is fetched, sent over the system bus, and received by the CPU.

There's no programming construct for dealing with caches, but if your code and data can fit neatly in the L1 cache, then it will be fastest. Next is if it can fit in the L2, and so on. If your code or data cannot fit at all, then you'll be at the mercy of the system memory, which can be orders of magnitude slower.

This is why counter-intuitive things like unrolling loops, which should be faster, might end up being slower because your code becomes too large to fit in cache. It's also why shaving a few bytes off a data structure could pay huge dividends even though the memory footprint barely changes. If it fits neatly in the cache, it will be faster.

The only way to know if you have a performance problem related to caching is to benchmark very carefully. Remember each processor type has varying amounts of cache, so what might work well on your i7 CPU might be relatively terrible on an i5.

It's only in extremely performance sensitive applications that the cache really becomes something you worry about. For example, if you need to maintain a steady 60FPS frame rate in a game, you'll be looking at cache problems constantly. Every millisecond counts here. Likewise, anything that runs the CPU at 100% for extended periods of time, such as rendering video, will want to pay very close attention to how much they could gain from adjusting the code that's emitted.

You do have control over how your code is generated with compiler flags. Some will produce smaller code, some theoretically faster by unrolling loops and other tricks. To find the optimal setting can be a very time-consuming process. Likewise, you'll need to pay very careful attention to your data structures and how they're used.

Admiral Grace Hopper used to hand out "light-nanoseconds" - 30 centimeter of string. A tiny eternity is about _10 meters_. — MSalters, Sep 18 '13 at 20:39

score 2 · Answer 2 · answered Sep 18 '13 at 18:26

[Cache] has different meaning in different contexts.

Bingo. Here are some definitions:

Cache

Verb

Definition: To place data in some location from which it can be more efficiently or reliably retrieved than its current location. For instance:

Copying a file to a local hard drive from some remote computer
Copying data into main memory from a file on a local hard drive
Copying a value into a variable when it is stored in some kind of container type in your procedural or object oriented program.

Examples: "I'm going to cache the value in main memory", "You should just cache that, it's expensive to look up"

Noun 1

Definition: A copy of data that is presumably more immediately accessible than the source data.

Examples: "Please keep that in your cache, don't hit our servers so much"

Noun 2

Definition: A fast access memory region that is on the die of a processor, modern CPUs generally have several levels of cache. See cpu cache, note that GPUs and other types of processors will also have their own caches with different implementation details.

Examples: "Consider keeping that data in an array so that accessing it sequentially will be cache coherent"

score 0 · Answer 3 · answered Sep 18 '13 at 18:17

My definition for Cache would be some thing that is in limited amount but faster to access as there is less area to look for. If you are talking about caching in any programming language then it means you are storing some information in form of a variable(variable is nothing a way to locate your data in memory) in memory. Here memory means both RAM and physical cache (CPU cache).

Physical/CPU cache is nothing but memory that is even more used than RAM, it actually stores copies of some data on RAM which is used by CPU very often. You have another level of categorisation after that as well which is on board cache(faster) and off-board cache. youu can see this link

The smaller size isn't generally the cause of the faster access. — Ben Voigt, Sep 18 '13 at 18:21

score 0 · Answer 4 · answered Sep 18 '13 at 19:36

I am updating the question because previously it was too broad. My confusion started with this answer. He says
RowData and m_data are specific to my implementation,
but they are simply used to cache information about a row in the file
What does cache in this context mean?

This particular use means that RowData is held as a copy in memory, rather than reading (a little bit of) the row from a file every time we need some data from it. Reading from a file is a lot slower [1] than holding on to a copy of the data in our program's memory.

[1] Although in a modern OS, the actual data from the hard-disk is probably held in memory, in file-system cache, to avoid having to read the disk many times to get the same data over and over. However, this still means that the data needs to be copied from the file-system cache to the application using the data.

What is cache in C++ programming?

4 Answers4