Some calls to "calloc" are suspiciously fast

Question

I'm benchmarking with “Perf” (Linux, gcc).

When allocating memory:

point_1 = calloc (100000000, 16);  //this takes nearly 1 second and perf find 27M transfers from RAM->CACHE and 1M from CACHE->RAM

This is OK.

But when trying to allocate two arrays:

point_1 = calloc (100000000, 16); 
point_2 = calloc (100000000, 16);
  //again, program takes nearly 1 second, 27M transfers RAM-CACAHE, 1M CACHE->RAM

It looks like, that second “calloc” (and all following) are behave like “malloc”. I'm using “gcc version 4.9.2 (Ubuntu 4.9.2-0ubuntu1~12.04) ”. Otherwise program works fine.

Is that behaving OK?

Here are some more tests and results:

Time for allocating of data structure 1: 0.976468 
  Perf: R:27M, W:1M

Time for allocating of data structure 1: 0.975402 
Time for initialization of data structure 1 to value of 7: 0.296787 
  Perf: R: 52M, W: 26M

Time for allocating of data structure 1: 0.976034 
Time for initialization of data structure 1 to value of 7: 0.313554 
Time for allocating of data structure 2: 0.000031   <-- misbehaving
  Perf: R: 52M, W:26M

Time for allocating of data structure 1: 0.975403 
Time for initialization of data structure 1 to value of 7: 0.313710 
Time for allocating of data structure 2: 0.000031   <-- misbehaving
Time for initialization of data structure 2 to value of 7: 0.809855 
  Perf: R:79M, W: 53M

As mentioned, everything works fine. No memory problems in Valgrind. — Athene Noctua, Jan 17 '15 at 18:59
In both cases, alloc + copying the value 7 takes about 1s. The system is simply more lazy the second time, it waits until you use the space to give it to you. — Marc Glisse, Jan 17 '15 at 19:10
If you are using `calloc()` how do you distinguish the timing between *allocating* and *initialization* of the data structure? — Weather Vane, Jan 17 '15 at 19:12
Ok, confirmed. Two large callocs, and the second is for some reason much faster. Let me see if I can find out something more. — Thomas Padron-McCarthy, Jan 17 '15 at 19:19
of course the OS can delay the heavy part of the work till there's access (which will cause a page fault). the interesting thing is: why? and why only for the second call? — Karoly Horvath, Jan 17 '15 at 19:22
related: http://stackoverflow.com/questions/2688466/why-mallocmemset-is-slower-than-calloc/2688522#2688522 — Karoly Horvath, Jan 17 '15 at 19:34
@weather Vane: “calloc” should initialized to 0. The second (special) initialization to 7 is simple for loop. — Athene Noctua, Jan 17 '15 at 19:37

score 3 · Answer 1 · answered Jan 17 '15 at 18:56

3

Each of the calls tries to allocate 1.6 GB of memory. I suspect the second call is failing, which would explain the symptoms. Check the return value from calloc().

answered Jan 17 '15 at 18:56

NPE

486,780
108
951
1,012

No. There is plenty of free memory. “calloc” returns something different then NULL. As you can see in additional tests (last one) initialization of second array is fine. – Athene Noctua Jan 17 '15 at 19:02
Then what is the problem? – i486 Jan 17 '15 at 19:05

score 2 · Accepted Answer · edited May 23 '17 at 11:57

2

It can confirm that the second calloc takes much shorter time. It seems that Linux decides to postpone some of the actual work.

On my system, the first calloc takes around 0.7 seconds. If I then iterate over the allocated memory area, setting it to something other than zero, this takes 0.2 seconds. In total, 0.9 seconds.

The second calloc then takes 0.0 seconds, but setting the second area takes 0.9 seconds. Same total time, but it seems that the second calloc, as Karoly Horvath wrote in a comment, doesn't actually create the memory pages, but leaves that to page faults when accessing the memory.

Another great comment by Karoly Horvath linked to this related question: Why malloc+memset is slower than calloc?

Tested on Ubuntu 14.04.1 LTS running on an Intel Core i7-4790K, with -O2 and a GCC that calls itself "gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2". Glibc version is Ubuntu EGLIBC 2.19-0ubuntu6.4.

edited May 23 '17 at 11:57

Community

1
1

answered Jan 17 '15 at 19:31

Thomas Padron-McCarthy

27,232
8
51
75

@Marc: True. Or perhaps the kernel? But I'm too incompetent to find which glibc version I am running. ldd --version says "ldd (Ubuntu EGLIBC 2.19-0ubuntu6.4) 2.19". Is that the right info? – Thomas Padron-McCarthy Jan 17 '15 at 19:37
@ThomasPadron-McCarthy, glibc usually is an executable itself, I get my version by running `/lib/i386-linux-gnu/libc.so.6`, for example. – mafso Jan 17 '15 at 19:40
2

Yes, that's it. You can also get it with `apt-cache policy libc6` or, for fun, by executing this command: `/lib/x86_64-linux-gnu/libc.so.6` (exact location may vary). – Marc Glisse Jan 17 '15 at 19:41
@Thomas & Karoly: Thanks for your tips. You are right. “page-faults” increases as predicted in article you suggested. But it does not answer the question why calloc behave's differently when called for the second time. I probably find the answer. Every single memory page, that OS gives to “malloc/calloc” is zeroed (security feature). Thats why calloc can be so fast. The problem is if “calloc” allocates memory already paged to this process (previously freed chunk of memory – or reserved in advance). In that case (probably) memset is called. – Athene Noctua Jan 18 '15 at 20:34
Additional reading can be found here: calloc-does-not-automatically-consumes-memory-out-of-ram . Comment from -indiv- and answer of -Zaibis- . (http://stackoverflow.com/questions/27002450/calloc-does-not-automatically-consumes-memory-out-of-ram) – Athene Noctua Jan 18 '15 at 20:37

Some calls to "calloc" are suspiciously fast

2 Answers2

Linked