What this 4K difference?

Question

I've writen a simple code for a simple test.

#include <stdio.h>

typedef struct
{
    void* Data;
}List;

void x()
{
    getchar();

    int i;
    List* myList[100000];

    for(i = 0; i < 100000; i++)
    {
        myList[i] = (List*)malloc(sizeof(List)*1024*1024);
    }

    getchar();

    for(i = 0; i < 100000; i++)
    {
        free(myList[i]);
    }
}

int main()
{
    x();
    getchar();
    return 0;
}

I've watched program in "taskmgr" in I've seen: - 540K - 4.500K - 544K (What's 4K difference?)

It's implementation dependent, and you haven't told us what implementation you're using. But anyway, probably better to just ignore this and move on--surely 4KB doesn't matter at all. — John Zwinck, Oct 17 '13 at 13:08

score 8 · Accepted Answer · edited May 23 '17 at 11:43

8

If all your allocations were to succeed, you would use about 100000 * 1024 * 1024 * sizeof (List) bytes, which is (assuming List is 4 bytes, 32-bit pointer) slightly over 390 GB.

Since you quote much lower numbers, it feels safe to assume that not all your allocations actually succeed. Actually, it's hard to imagine them all succeeding on a 32-bit system ... Anyway, your code never checks this, so it's hard to tell.

Lesson: malloc() can fail, and if you don't check the return value you can't assume that the allocation has succeeded.

Also, don't cast the return value of malloc() in C.

edited May 23 '17 at 11:43

Community

1
1

answered Oct 17 '13 at 13:08

unwind

391,730
64
469
606

1

Another important point is that `malloc()` can _succeed_. Even if you're allocating a hundred times more RAM than the system physically has. And then not actually commit the pages until you try to use them. (I don't know if Windos does this, though.) – This isn't my real name Oct 17 '13 at 14:56

score 1 · Answer 2 · answered Oct 17 '13 at 13:17

1

Modern operating systems tend to overcommit resources and actually perform allocation only when you make use of the resources. This is done for memory using paging and page faults. Since you never read from or write to the memory you requested via malloc(), it's more than likely the operating system never mapped any of the pages.

Try writing to each allocated buffer. Reading might not be enough because it's possible to map a system- or process-wide zero page and rely on Copy On Write (COW) to perform the actual allocation. This is often the case.

Even then, if you allocated more than the length of a page for each buffer, you might need to perform a write per-page to actually get all the pages.

Do that and watch your process grow much fatter.

answered Oct 17 '13 at 13:17

idoby

907
8
20

Just a semi-related question. Where does this concept of a "zero page" actually come from? I have seen the VM code of many operating systems, I've worked on a few of them and I've never seen anything like that implemented, never saw the point of it (it could be quite inefficient for a bunch of reasons) and never even saw it mentioned in literature except people talking about it on the internet. The closest to a zero page is for simplifying the implementation of reading of /dev/zero on various unix-like systems (it's not related to mmap of /dev/zero though, don't mix that up). – Art Oct 17 '13 at 14:01
It's an optimization that enables the OS to supply zeroed pages quickly, for instance, to replace `memset(..., 0)` calls or when loading the BSS segment of a program binary. Also, if you check, you'll probably notice that most new memory given to you by Linux (say via `malloc()`) is zeroed because it's easiest to map the zero page and actually perform the allocation via COW later. You'll also find references to this concept in the Linux source. – idoby Oct 18 '13 at 06:44
It's not an optimization. It's a suboptimization on any modern machine because of the TLB invalidation cost when you do calloc->read->write or something like that (or for that matter read BSS), which isn't uncommon. I see you're right though. At least in early linux 2.6 they used to do something crazy like this, but they don't seem to anymore, at least not in an obvious way, now at least I know where the notion comes from and my curiosity is satisfied. – Art Oct 18 '13 at 11:55

What this 4K difference?

2 Answers2