1

In C, I'm trying to create an integer array with 1,000,000,000 (1 billion) elements. I tried to use;

#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <string.h>
#include <math.h>
#include <time.h>

#define population 1000000000

int numberOfHandshaking;

int main(int argc, char **argv) {
    printf("******************************\n");
    printf("     FRIEND CIRCLE QUERIES\n");
    printf("******************************\n\n");

    FILE *fPointer;
    fPointer = fopen("C:\\Users\\ASUS\\Desktop\\AdvProTech2\\uebung_6\\Ass13in.txt", "r");

    fscanf(fPointer, "%d", &numberOfHandshaking); // Reads the first line
    printf("numberOfHandshakings: %d\n", numberOfHandshaking);

    int *persons = malloc((population + 1) * sizeof(int));

    for (int i = 0; i < 127; i++) {
        printf("persons[%d]= %d \n", i, persons[i]);
    }

    printf("\n\n\n");
    return 0;
}

it does not give error when compiling but crashes when trying to reach the elements. Can anyone help me?

chqrlie
  • 131,814
  • 10
  • 121
  • 189
jensoncan
  • 49
  • 1
  • 6
  • 1
    homeworks requires that the population is 10^9 – jensoncan Apr 14 '20 at 18:38
  • Unless your memory supply has at least that much contiguous memory, `malloc` will always return NULL, and if you do not handle that condition, yes, your program will likely crash. – ryyker Apr 14 '20 at 18:42
  • 1
    That's a hint that you shouldn't be allocating an array, and instead there is a better algorithm to solve it. – Blindy Apr 14 '20 at 18:42
  • @ryyker ok i understood the problem. But still, i need to allocate that much memory. – jensoncan Apr 14 '20 at 18:45
  • @Blindy can you at least post such a solution. I am really new in C language. – jensoncan Apr 14 '20 at 18:46
  • 1
    you're probably interested in something like a sparse array, where all the elements theoretically exist, but in practice only the ones that get filled in use the full memory size of the data type – Fubar Apr 14 '20 at 18:47
  • A solution of what? I have no idea what homework you're looking at. And I told you how you can allocate that much memory a little below, if you insist on going the wrong way about it. – Blindy Apr 14 '20 at 18:47
  • There is an [example project here](https://codereview.stackexchange.com/questions/162546/send-command-and-get-response-from-windows-cmd-prompt-silently) that has been used to read a buffer well over that size, but in smaller bite segments. Look at it and see if you can use parts of it to help you understand. It is in ANSI C, compiled in GCC and CLANG without issue. – ryyker Apr 14 '20 at 18:50
  • 1
    It is likely your homework assignment can be solved without allocating an array that big. Show the assignment. – Eric Postpischil Apr 14 '20 at 18:50
  • @Fubar yes maybe the homework can be done in this way. But i wanted to have all the population in an array. – jensoncan Apr 14 '20 at 18:57
  • @EricPostpischil you can have a look at this; The population of HackerWorld is 10^9. Initially, none of the people are friends with each other. In order to start a friendship, two persons a and b have to shake hands, where 1 < a, b < 10^9. The friendship relation is transitive, that is if a and b shake hands with each other, a and friends of a become friends with b and friends of b. You will be given q queries. After each query, you need to report the size of the largest friend circle (the largest group of friends) formed after considering that query. – jensoncan Apr 14 '20 at 18:59
  • `int *persons = malloc((1000000000+1) * sizeof(int));` should include a test for success: `int *persons = malloc((1000000000+1) * sizeof(int)); if(persons) {use it} else return 0;` – ryyker Apr 14 '20 at 19:01
  • @ryyker i already did that as i wrote in the first post. It does not work :( – jensoncan Apr 14 '20 at 19:05
  • As `1000000001 * 4` bytes are less than 4 Gibibytes I would presume that the problem might rest somewhere else in your code (Yes, I am aware of how different implementations of`malloc` work and that it might be still a `malloc` problem). Please post it all. Assuming, of course, that it compiled without a warning and all warnings have been switched on. – deamentiaemundi Apr 14 '20 at 19:05
  • @deamentiaemundi I edited the first post – jensoncan Apr 14 '20 at 19:11
  • @deamentiaemundi: Many 32-bit systems limit the maximum allocation of a program to considerably less than 4 Gibibytes. For example on 32-bit Linux the limit is about 3 Gibibytes, see https://stackoverflow.com/questions/5079519/memory-limit-to-a-32-bit-process-running-on-a-64-bit-linux-os. And of course it's also limited by the amount of physical memory and swap that the system actually has. – Nate Eldredge Apr 14 '20 at 19:13
  • This appears to be [this problem](https://www.hackerrank.com/challenges/friend-circle-queries/problem). q is limited to 100,000. An array of a billion elements is not needed. Think of another way to solve the problem. – Eric Postpischil Apr 14 '20 at 19:27
  • I’m voting to close this question because it is an X-Y question. The poster asked how to solve a sub-problem that was not necessary or appropriate for their actual problem. – Eric Postpischil Apr 14 '20 at 19:28
  • @NateEldredge Yes, I'm aware of all that, but OP said the malloc-check had been implemented. It was more of a "that's not that much" comment but the fact that the constant was just less than 4GB made it a bit ambiguous, admitted. I promise to be more careful next time. But with the complete code given it is more likely that the problem lies…uhm…elsewhere. – deamentiaemundi Apr 14 '20 at 19:31

3 Answers3

2

When working with ridiculous numbers like these, compile your program in 64 bits. Or simply write better code, because I can guarantee you that you don't need that array allocated like that.

Blindy
  • 65,249
  • 10
  • 91
  • 131
  • and how can i compile in 64 bits? – jensoncan Apr 14 '20 at 18:37
  • 1
    @jensoncan, depends on your compiler. `gcc` uses `-march=x64`, `cl` uses a cross-compiler pattern where you select your host architecture (probably `x64`) and your target platform (`x64`) and choose the compiler sitting, for example, at `C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.25.28610\bin\Hostx64\x64` – Blindy Apr 14 '20 at 18:40
  • if he is not running on a 64bitt machine (It seems, as he is running windows) if windows had detected a 64bit compatible architecture, then it should have installed a 64bit version. – Luis Colorado Apr 16 '20 at 06:15
  • @jensoncan, 64bit is not something you can select at compile time, you need a 64bit hardware in order to be able to run it. There is no sense in compiling (forcing the compiler to compile it 64bit) if you cannot run it in your computer. – Luis Colorado Apr 16 '20 at 06:17
  • You don't know what you're talking about, please don't spread misinformation where newbies are concerned. – Blindy Apr 16 '20 at 14:29
1

First of all, you don't say in which architecture you are trying this approach. Second, you don't say what operating system you are using. You only say you are trying to malloc(3) an array of one billion ints.

Think twice: a single array of 1 000 0000 001 int elements is 4 000 000 004 in size. If you are trying to handle this in a 32 bit architecture, you are asking the system to allocate the full virtual memory space (which is 4Gb) just for your array. Most probably you have a very hard limit that impedes you to allocate such amount of memory.

In 64bit architecture you have enough addresses to handle that, but again, probably your operating system will refuse to create a single contiguous mapping of such size (or the amount of virtual memory your system can handle). The most probable thing is that you'll not be able to get that memory. I see you are running this in windows (by the path you are using for the file name) so most probably you cannot even check how much memory you can handle per process.

I have tested your program in a FreeBSD 64bit architecture, in which I have the following limits (I have to admit I have tweaked them a little to handle your problem for my user account, as a normal user in a multiuser system is never granted such limits, as default):

$ ulimit -a
number of pseudoterminals            (-P) unlimited
socket buffer size       (bytes, -b) unlimited
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 33554432
file size               (blocks, -f) unlimited
max kqueues                     (-k) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 230121
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 524288
cpu time               (seconds, -t) unlimited
max user processes              (-u) 12042
virtual memory          (kbytes, -v) unlimited
swap size               (kbytes, -w) unlimited

(as you can see, I have a maximum data segment size limit of 33Gb, so, in case I had asked for 10 billion elements array, I had failed, and need to say that that limit is imposed by the kernel, I cannot raise it --even as root)

  • When I did this, I got a SIGSEGV because the file pointer returned by fopen(3) gave NULL (the file didn't exist, another thing you don't check in your program) so the program was aborted on the fscanf(3) call. Note: You need to check the return values of calls for errors.
  • After that, I was perfectly able to run (and indeed to allocate a billion --plus one-- elements array) without any problem. FreeBSD program runs in 0.001s from beginning to end, as it doesn't have to allocate all that memory in one chunk (indeed, it doesn't have to allocate all, as you only use the first 126 elements of the array, so indeed, you will allocate only one page of such memory, 504 bytes of memory.
$ pru
*****************************************
     THIS WAS FRIEND CIRCLE QUERIES
*****************************************

numberOfHandshakings: 99885
persons[0]= 0 
persons[1]= 0 
persons[2]= 0 
[...]
persons[123]= 0 
persons[124]= 0 
persons[125]= 0 
persons[126]= 0 



Then I fixed a maximum virtual memory space of 4Gb, and repeated the test:

$ pru
*****************************************
     THIS WAS FRIEND CIRCLE QUERIES
*****************************************

numberOfHandshakings: 99885
malloc failed: Cannot allocate memory

this time, the result to ulimit -a was:

$ ulimit -a
number of pseudoterminals            (-P) unlimited
[... same as before ]
max user processes              (-u) 12042
virtual memory          (kbytes, -v) 4194304 <<<<<< 4Gb virtual memory.
swap size               (kbytes, -w) unlimited

Your program, after the modifications I did on it to be able to process the errors returned by the calls to fopen() and malloc() was:

#include <ctype.h>
#include <errno.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#define population 1000000000

int numberOfHandshaking;

int main(int argc, char **argv)
{
    printf("*****************************************\n");
    printf("     THIS WAS FRIEND CIRCLE QUERIES\n");
    printf("*****************************************\n\n");

    FILE *fPointer;
    fPointer =
        fopen("Ass13in.txt", /* I changed this, my apologies */
            "r");

    if (fPointer == NULL) {
        printf("couldn't open file: %s\n", strerror(errno));
        exit(1);
    }

    fscanf(fPointer, "%d", &numberOfHandshaking); // Reads the first line
    printf("numberOfHandshakings: %d\n", numberOfHandshaking);

    int *persons = malloc((population+1) * sizeof(int));

    if (persons == NULL) {
        printf("malloc failed: %s\n", strerror(errno));
        exit(1);
    }

    for (int i=0; i<127; i++){
        printf("persons[%d]= %d \n", i, persons[i]);
    }

    printf("\n\n\n");
    return 0;
}

and it run fine!! :)

NOTE:

In your program you #include files that you don't need to. Those are:

#include <ctype.h>
#include <math.h>
#include <time.h>
Community
  • 1
  • 1
Luis Colorado
  • 10,974
  • 1
  • 16
  • 31
0

If you do not know the needed size of a buffer before runtime, smaller bites are preferable when allocating memory than very very large chunks, where the likelihood of failing due to lack of contiguous memory available is high.

Start off with calloc/malloc, then use realloc:

// return '*str' after number of bytes realloc'ed to 'size'
static char * ReSizeBuffer(char **str, unsigned int size)
{
    char *tmp=NULL;

    if(!(*str)) return NULL;

    if(size == 0)
    {
        free(*str);
        return NULL;
    }

    tmp = (char *)realloc((char *)(*str), size);
    if(!tmp)
    {
        free(*str);
        return NULL;
    }
    *str = tmp;

    return *str;
}
ryyker
  • 22,849
  • 3
  • 43
  • 87