Why am I getting segfault from this unsigned int?

Question

I'm trying to initialize an integer array and set all elements to 1. I need the array to have an upper bound of 4294967295, or the maximum number possible for a 32-bit unsigned int.

This seems like a trivial task to me, and it should be, but I am running into segfault. I can run the for loop empty and it seems to work fine (albeit slowly, but it's processing nearly 4.3 billion numbers so I won't complain). The problem seems to show up when I try to perform any kind of action within the loop. The instruction I have below - primeArray[i] = 1; - causes the segfault error. As best as I can tell, this shouldn't cause me to overrun the array. If I comment out that line, no segfault.

It's late and my tired eyes are probably just missing something simple, but I could use another pair.

Here's what I've got:

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <stdint.h>

#define LIMIT 0xFFFFFFFF;

int main(int argc, char const *argv[])
{
    uint32_t i;

    uint32_t numberOfPrimes = LIMIT;        // hardcoded for debugging
    int *primeArray = (int*) malloc(numberOfPrimes * sizeof(int));

    for (i = 0; i < numberOfPrimes; ++i) {
        primeArray[i] = 1;
    }
}

(a) is this on a 32bit system? (b) regardless, are you aware you just asked for 16gB (or 32gB depending on your `int` depth) of RAM to be allocated? (c) any particular reason you didn't check the result of that `malloc()` before using it? (d) if this is going to be an [Sieve of Eratosthenes](http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes), do you understand you don't need a full-on `int` for the data member, a *bit* will suffice, and a `unsigned char` is the easiest thing to code? — WhozCraig, Mar 10 '13 at 08:42
@WhozCraig I'm operating on a 64-bit system, and yes, I'm aware of the resources I'm requesting. I have enough memory that I should be covered, but it's possible that I'm overestimating how much is available to me at the moment. I don't intend on using the full scope for the final product, only to test my capabilities right now. I will give serious consideration to switching to a bit, per your suggestion. Thanks! — idigyourpast, Mar 10 '13 at 08:59
Just use an `unsigned char` first. It will be *much* simpler (and faster) to code the algorithm. Believe me. — WhozCraig, Mar 10 '13 at 09:00
@idigyourpast Even if you have 64gigs of RAM installed (do you?), the OS won't give all of it in its entirety to one process. The OS/the kernel ain't a politician - it tries to maintain true democracy between processes throughout the lifetime of the system. — , Mar 10 '13 at 09:00
@H2CO3 Haha no, unfortunately I'm not quite that lucky. I'm still learning about operating systems/kernels, so thank you for the heads up on that. — idigyourpast, Mar 10 '13 at 09:08
@idigyourpast This is about the [simplest sieve](http://ideone.com/aqsYo9) example I can think of. The only reason I'm not a [`std::bitset`](http://en.cppreference.com/w/cpp/utility/bitset) fan is the requisite sizing at *compile* time rather than runtime. You can get creative though, using an array of them and some creative looping, but for simple sieves I rarely bother. Yours is pretty big, however, so consider it. — WhozCraig, Mar 10 '13 at 09:20
I don't understand. If compiled for 64-bit and run on a 64-bit system with sufficient RAM/pagefile, why should this not work?? — Martin James, Mar 10 '13 at 09:31
@idigyourpast malloc allocates continuous block of memory , you may have enough ram , but do you have enough contiguous ram ?? — Barath Ravikumar, Mar 10 '13 at 09:32
@MartinJames , yes you may have sufficient ram , but getting sufficient continuous RAM more than a certain ammount , is nearly impossible , that is because , over time several processes move in and out of memory , and often create unusable wasted segments of memory , meaning though you may have 16gigs of ram , the bits and pieces of hundreds of programs are scattered all over the place , this becomes worse at time progresses , just read about fragmenation for starters http://en.wikipedia.org/wiki/Fragmentation_(computing) — Barath Ravikumar, Mar 10 '13 at 09:45
@BarathBushan Fragmentation applies to a single process, not to the OS. The kernel manages mapping of virtual memory to physical memory, so it can provide a process 16GB of "contiguous" virtual memory, even though that much contiguous physical memory is not available. Just try allocating a 6GB chunk on an 8GB machine, it works just fine despite other processes "fragmenting" the memory. — user4815162342, Mar 10 '13 at 11:48
In fact, allocating memory in many small chunks is the **cause** of fragmentation, not a cure. — user4815162342, Mar 10 '13 at 11:49
"The kernel manages mapping of virtual memory to physical memory, so it can provide a process 16GB of "contiguous" virtual memory" . sorry to break it to you sir , you have never been so wrong lately.....Its not the "kernel" mapping , it is the Virtual memory manager (VMM) , which maps the logical pages(not physical) to virtual adresses ,and it does not provide 16 GB of memory to a process , all it does is provide 16 GB of "address space" not the actual memory , if the process continues to consume all the 16 gb , first page stealing will be a consequence , later a process "thrashing" is done — Barath Ravikumar, Mar 10 '13 at 12:02
@BarathBushan Sure, but the kernel manages the VMM, and it's the kernel's responsibility to maintain the mapping. The point is that the process can get its "contiguous" memory regardless of OS-level fragmentation, so the OS's failure to allocate has zero to do with "fragmentation". If you don't believe me, just try it. — user4815162342, Mar 10 '13 at 13:19
@BarathBushan: Address space fragmentation only happens within the virtuall address space of each process. On the system level memory is managed in equally sized pages, which may be scattered all over the place and virtual contigous memory is likely scattered in noncontigous pages. Most OS will make an effort to move pages around to make them contigous (to enhance caching efficiency); and as long as there's one free page around (about 2kiB to 64kiB depending on system configuration) it can do this without problem. — datenwolf, Mar 10 '13 at 15:06
@BarathBushan: Also on most modern OS you can allocate way more virtual address space, than there is physical memory. This is again due to paging: Only when something is written to address space, a page needs to be backed by memory. So even if you allocate 128GiB of address space, as long as you write to less page sized blocks in it, than there are system memory backed pagings you're fine. — datenwolf, Mar 10 '13 at 15:09
@datenwolf , fair point made , but if fragmentation cannot explain the reason malloc() fails to return a large contiguous block of memory , do you have any other theory , supporting malloc's failure ?? — Barath Ravikumar, Mar 10 '13 at 16:20
@BarathBushan: malloc doesn't return memory. It returns address space. And the OS may be configured to make malloc return at most a certain fraction of the installed system memory size in address space. The default value of most Linux installations is 50%. If you ask for more, you'll get back nothing. However using other allocation methods (like mmap) it's possible to circumvent even this limit (unless several memory safeguards have been enables, usually only the case in high availability systems). — datenwolf, Mar 10 '13 at 16:41
@BarathBushan: Also take not that it is possible to entirely disable memory overcommitment. In this case all address space requested must be backed by a memory allocation. Most high availability systems are configured this way. For example the server hosting my websites and email system is configured to do no memory overcommitment. — datenwolf, Mar 10 '13 at 16:43
@datenwolf , thanks for your time , i really got the hang of it now. — Barath Ravikumar, Mar 10 '13 at 16:56

score 10 · Accepted Answer · answered Mar 10 '13 at 08:43

10

Check the return code from malloc() to make sure the array was actually allocated. I suspect that the following test would fail:

int *primeArray = (int*) malloc(numberOfPrimes * sizeof(int));

if (primeArray != NULL) {  /* check that array was allocated */
    for (i = 0; i < numberOfPrimes; ++i) {
        primeArray[i] = 1;
    }
}

answered Mar 10 '13 at 08:43

chrisaycock

36,470
14
88
125

3

casting malloc ?? Read this http://stackoverflow.com/questions/605845/do-i-cast-the-result-of-malloc – Barath Ravikumar Mar 10 '13 at 08:47
@BarathBushan The cast isn't why the OP's program is crashing. – chrisaycock Mar 10 '13 at 08:50
1

@chrisaycock Thank you, I had forgotten to check that. And you're right, it isn't allocating. – idigyourpast Mar 10 '13 at 08:54
2

@chrisaycock [Sometimes it is](http://stackoverflow.com/questions/7545365/why-does-this-code-segfault-on-64-bit-architecture-but-work-fine-on-32-bit), so casting is a bad piece of advice. – Mar 10 '13 at 08:59
1

@chrisaycock , yes the bug may not be active right now , but if encouraged , it will show up someday , somewhere , so better crush the bug right now... – Barath Ravikumar Mar 10 '13 at 09:00
@BarathBushan Thank you for posting that link. I'll be following that closely. I've had a lot of people tell me to cast malloc, so this is welcome and interesting news. – idigyourpast Mar 10 '13 at 09:06
@idigyourpast Tell those people to learn C before trying to teaching it :) – Mar 10 '13 at 09:09

user4815162342 · Answer 2 · 2013-03-10T14:57:50.520

5

Your malloc call requests 16 gigabytes of memory from the system. If you don't have that much free virtual memory, or if you are running on any 32-bit system, the call will fail. If you don't check for the failure of malloc, as your code doesn't, the array will be NULL and any subsequent access to its elements will cause a segmentation fault.

If you really need to work with an array that large, you will either need to get a 64-bit system with a lot of memory, or rewrite your program to work with a smaller working set, and persist the rest to disk.

edited Mar 10 '13 at 14:57

answered Mar 10 '13 at 08:50

user4815162342

141,790
18
296
355

Regarding the 32 bit system situation: Yes. But on a 64 bit system malloc may return non-null even if there's less than 16GiB of memory (RAM+swap) installed. Modern OS allocate address space not memory. Only when something is written to address space, a piece of memory, called a page is allocated to back the page sized chunk of address space, to which was written. In the case of OP it may simply happen that the system eventually runs out of allocatable pages. – datenwolf Mar 10 '13 at 15:01
@datenwolf The system can certainly overcommit, but `malloc` succeeding when requesting space that *exceeds* RAM+swap sounds like an OS bug. Either way, there is no way to portably test for that situation, so one has no choice but to be optimistic and assume that, when `malloc` succeeds, the memory is available. That's how most software works. – user4815162342 Mar 10 '13 at 15:28
Ideed most OS will set default malloc size limit at about half the available system memory. However most OS allow to change this configuration to allow malloc allocations exceeding available system memory. – datenwolf Mar 10 '13 at 15:31

Why am I getting segfault from this unsigned int?

2 Answers2