3

I wrote a C program to merge-sort (recursive) integers, using a dynamically allocated array. It works fine with up to 100k integers, but when I feed 1 million integers, it throws the Segmentation fault (core dumped) error.

Why does it do this? Is my 16GB RAM not good enough? Would it be able to sort bigger number of integers if I didn't use a dynamically allocated array?

How does dynamic allocation exactly work? From my understanding, when a dynamic variable or an element in an dynamic array is declared, a portion of memory (RAM?) is put aside and set to strictly store the declared variable until the memory is freed.

When my program tries to set aside memory to hold a million integers, does it fail because there isn't enough memory available?

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#define BIL 1E9

//struct Sort allows dynamic allocation of the array used in insertion sort.
typedef struct {
    int *arr; //pointer to the dynamic array
    size_t used; //stores number of 'used' elements in the array
    size_t size; //stores number of total elements
} Sort;

//function prototypes to interact with the dynamic array
void freeSort(Sort *);
void initSort(Sort *, size_t);
void inSort(Sort *, int);

//prototypes for the Merge Sort
void mergeSort(Sort *, int, int, int []);
void merge(Sort *, int, int, int, int []);
void copyArray(int [], int, int, Sort *);



int main(){

    //declare Sort variable 'magic' to perform the magical insertion sort on the dynamic array.
    Sort magic;
    initSort(&magic, 10); //initialize magic with 10 elements

    //variables to allow the program to function
    int intin;
    char filename[15];

    //tosort is the file to sort.
    //sorted is the output file after sort.
    FILE *tosort, *sorted;

    //necessary variables to measure time
    struct timespec start, finish;

    //prompt user for file name.
    printf("Enter the name of file with a list of integers to sort: ");
    scanf("%s", filename);
    tosort = fopen(filename, "r"); //read 'tosort' file

    //write the 'sorted' file to 'filename.sorted'
    sorted = fopen(strcat(filename, ".sorted"), "w");

    //while loop stores every integer in the dynamically allocated magic array from tosort file.
    while (!feof(tosort)) {
        fscanf(tosort, "%d", &intin);
        inSort(&magic, intin);
    }

    //n stores number of integers to sort
    int n = magic.used;
    //temporary array for use with the merge sort
    int sortedArray [n];

    //measure time
    clock_gettime(CLOCK_REALTIME, &start);  //start

    //Merge Sort
    mergeSort(&magic, 0, n, sortedArray);

    clock_gettime(CLOCK_REALTIME, &finish); //finish

    //calculate the elapsed time in nanoseconds.
    double elapsed = (finish.tv_sec-start.tv_sec)+(finish.tv_nsec-start.tv_nsec)/BIL;

    printf("Merge Sort took %lf seconds\n", elapsed);

    //write the sorted array to 'sorted' ('filename'.sorted)
    for (int i = 0; i < n; i++) {
        fprintf(sorted, "%d\n", magic.arr[i]);
    }

    //free up the allocated memory for the Sort array and close the files.
    freeSort(&magic);
    fclose(tosort);
    fclose(sorted);

    return 0;
}

//initialize the dynamic array
void initSort(Sort *dynA, size_t initSize) {
    dynA->arr = (int *)malloc(initSize * sizeof(int));
    dynA->used = 0;
    dynA->size = initSize;
}

//add values to the elements of the dynamic array
void inSort(Sort *dynA, int val) {
    //if the array size is not big enough to fit new values, allocate 100 more elements.
    if (dynA->used == dynA->size) {
        dynA->size += 100;
        dynA->arr = (int *)realloc(dynA->arr, dynA->size * sizeof(int));
    }
    //'used' holds the number of used elements with values in the array.
    dynA->arr[dynA->used++] = val;
}

//free allocated memory for the dynamic array
void freeSort(Sort *dynA) {
  free(dynA->arr);
  dynA->arr = NULL;
  dynA->used = dynA->size = 0;
}


//split the array until size is 1
void mergeSort(Sort *dynA, int begin, int end, int tempA [])
{
    //if size is 1, done splitting.
    if(end-begin < 2)
        return;

    // recursively split the array
    int mid = (end+begin)/2; // mid = middle point
    mergeSort(dynA, begin, mid, tempA); // mergeSort left half
    mergeSort(dynA, mid, end, tempA); // mergeSort right half
    merge(dynA, begin, mid, end, tempA); // merge the two halves
    copyArray(tempA, begin, end, dynA); // copy the merged array to dynA
}

//merge the two arrays
void merge (Sort *dynA, int begin, int mid, int end, int tempA [])
{
    int i = begin; int j = mid;

    //from begin to end, compare the values of the two arrays
    for (int k = begin; k < end; k++)

        // store the smaller value into tempA[k]
        if (j >= end || (i < mid && dynA->arr[i] <= dynA->arr[j]))
            tempA[k] = dynA->arr[i++];
        else tempA[k] = dynA->arr[j++];
}

//copy the contents of the temporary array to the dynamic array
void copyArray(int tempA[], int begin, int end, Sort *dynA){
    for(int k = begin; k < end; k++)
        dynA->arr[k] = tempA[k];
}

Cygwin64 and CommandPrompt give same faults when I feed a million integers to sort.

James
  • 51
  • 7
  • 2
    (1) Please try running a debugger and collect the backtrace. (2) Please check the return values of `malloc()` and `realloc()`. (3) Please show the call(s) to `inSort()`. – Arun Oct 05 '16 at 22:02
  • @Arun Here's the whole thing. I don't know how to use a debugger yet. – James Oct 06 '16 at 06:48
  • 1
    The way you're using `strcat` with `filename` leads to a buffer overflow. – Michael Foukarakis Oct 06 '16 at 09:22
  • @MichaelFoukarakis How so? Is it because the function calls are nested? `fopen(strcat(filename, ".sorted"), "w");` This does look like a 'poor style' of code. Though the program runs OK with a smaller number of integers to merge-sort. – James Oct 06 '16 at 11:52
  • 1
    @James, simply put, `filename` does not have enough space for the result of `strcat` when you input a filename of more than 7 characters. – Michael Foukarakis Oct 06 '16 at 12:28
  • when calling any of the `scanf()` family of functions, always check the returned value (not the parameter value(s)) to assure the operation was successful. When using the `%s` format specifier with any of the `scanf()` family of functions, ALWAYS use a `max characters` modifier that is one less than the length of the input buffer, to avoid any buffer overflow, which would be undefined behavior and can lead to a seg fault event. – user3629249 Oct 06 '16 at 19:31
  • *I don't know how to use a debugger yet.* This will/is a major hindrance to your becoming a good programmer, or even to debug your own code. Strongly suggest learning your IDE debugger and/or the `gdb`/`ddd` debugger – user3629249 Oct 06 '16 at 19:34
  • when working with large amounts of data, do NOT put the data on the stack, use `mmap()` or `malloc()` to obtain the needed room for the data. – user3629249 Oct 06 '16 at 19:36
  • *How does dynamic allocation exactly work?* The details (normally) do not matter. The key factor is that when the program is loaded, a huge amount of memory is allocated the 'heap'. Which is usually implemented as linked lists. Each time any of the heap memory allocation functions (malloc, calloc, realloc) is called, a section of that heap becomes a separate entry in the linked list. Then a pointer to the data part is returned to the program. 1) do not change that pointer, 2) do not write beyond that end of that data area 3) , call free() with that pointer to cleanup. – user3629249 Oct 06 '16 at 19:43
  • do not use `feof()` to control a loop. It does not do what you are expecting. use the call to `fscanf()` to control the loop. – user3629249 Oct 06 '16 at 19:50
  • you haven't shown any example of the contents of the input file, so we are not sure that the call to `fscanf()` is the correct function to use. – user3629249 Oct 06 '16 at 19:51
  • when calling the function: `fopen()`, always check (!=NULL) the returned value to assure the operation was successful. – user3629249 Oct 06 '16 at 19:55
  • for ease of readability and understanding, follow the axiom: *only one statement per line and (at most) one variable declaration per statement.* – user3629249 Oct 06 '16 at 19:58
  • in C, the returned type from the memory allocation functions is `void *` so can be assigned to any other pointer. Casting the returned type just clutters the code, making it more difficult to understand, debug, maintain. – user3629249 Oct 06 '16 at 20:01
  • when calling `realloc()`, always set a temporary variable, then check (!=NULL) that temporary variable, before setting the 'target' variable Otherwise, when `realloc()` fails, the 'target' variable will contain NULL, so the pointer to the previously allocated memory is lost, resulting in a memory leak. And accessing where the modified target variable points will result in a seg fault event. – user3629249 Oct 06 '16 at 20:10
  • the function: `mergeSort()` is using recursion. The depth of that recursion will be about log2(n) where n is the size of the incoming data array. Such a recursion depth may exceed the limits of the `stack` size, resulting in a seg fault event. – user3629249 Oct 06 '16 at 20:19

3 Answers3

3

Your bug is that you are using a large stack based VLA sortedArray. With 1,000,000 values, your array is 4MB and you get a segfault due to stack overflow because the array exceeds the preset stack size limit.

For example, under linux, the stack limit is around 8MB [on my system, I had to increase the array count to 3,000,000 to reproduce the segfault]


Change:

    //temporary array for use with the merge sort
    int sortedArray [n];

Into:

    //temporary array for use with the merge sort
    int *sortedArray = malloc(sizeof(int) * n);

Optionally, at the bottom of main you can add a free(sortedArray) for tidiness if you wish.


Also, you can use the limit shell command to see what the stacksize parameter is set to. So, an alternate way to fix the problem would be to use the command to increase the limit. But, I do not recommend that as it's a less general and more fragile solution than simply using malloc.


On debugging tips ...

To find your problem, I did the following:

Compiled the program with debug info: gcc -o blah -g blah.c

Invoked the debugger with gdb ./blah

Ran the program [inside the debugger] with run

I got the segfault information below:

Starting program: ...
Enter the name of file with a list of integers to sort: input2

Program received signal SIGSEGV, Segmentation fault.
0x00000000004009d1 in main () at blah.c:64
64      clock_gettime(CLOCK_REALTIME, &start);  //start

It didn't make much sense that clock_gettime was faulting, so I looked at the line above it:

    int sortedArray [n];
Craig Estey
  • 30,627
  • 4
  • 24
  • 48
  • Thanks! I can now sort a multiple million integers and understand stack better than before. – James Oct 06 '16 at 12:49
  • Nice catch. Just a quick tidbit: I believe that the UNIX command in question is `ulimit` not `limit` iirc? – LambdaBeta Oct 06 '16 at 14:48
  • @LambdaBeta Yes. To be effective, the command has to be a shell builtin. I use `tcsh` for my shell and the command is `limit`. For `bash` et. al. the command is `ulimit`. I had remembered this, later, but was too tired to add it. Besides, I don't condone use of `bash` :-) Seriously, the programmatic way would be to use `getrlimit/setrlimit` within the target program – Craig Estey Oct 06 '16 at 19:37
0

You are looking in the wrong place. Your heap memory is fine, the issue is your stack. Every time you call a function in C (unless it is compiled away or inlined) the compiler 'saves' all the local variables of the current function (including arguments) then calls the next function so that the previous function's information is ready when it ends. This is the stack.

By recursively calling mergesort on a million values, you not only allocate about 4MB on the heap, you also allocate sizeof(ALL_THE_STUFF_MERGE_SORT_NEEDS) * 1000000 bytes on the stack. This will almost certainly produce a stack overflow.

Try unraveling the recursion in mergesort to use a loop instead. That way you don't have to 'save' the function's state each recursive call and can just re-use the same variables over again. (You can google search for unrolled mergesort online to see how it works)

EDIT: I forgot that mergesort will only call itself 20 times if properly implemented on a million integers, nonetheless this may still be the problem, or at least related to it, so I'll keep my answer here.

LambdaBeta
  • 1,479
  • 1
  • 13
  • 25
  • 1
    Actually, the [max] number of stack frames is only `log2(1000000)`, which is ~20. – Craig Estey Oct 05 '16 at 21:09
  • Good point, although the merge at the 'leaf' level could have also been implemented recursively. I wonder if there's a memory leak in the mergesort itself, or if the algorithm is using up a large amount of stack memory. I guess we'd need the source to know for sure. – LambdaBeta Oct 05 '16 at 21:13
  • Yes. Except for the missing checks for `NULL` return from `malloc/realloc` [unlikely based on the stated 1000000 and 16GB], the error would seem to be in the code that has not [yet] been posted. – Craig Estey Oct 05 '16 at 21:19
  • @CraigEstey Edited the post and included the whole code. – James Oct 06 '16 at 06:49
0

Your array allocator seems OK. The problem is elsewhere:

  • which implementation of mergesort do you use?
  • can you post the code?
  • does it allocate memory for the temporary working array on the stack or from the heap?
  • if this temporary array is allocated on the stack (automatic storage), it is very likely the cause for your problem as the stack space is usually limited by default to a few megabytes on current systems.

EDIT: the temporary array is indeed allocated in automatic storage (on the stack) in the main() function: int sortedArray[n];. Allocating this with malloc() should solve the issue, but you have other issues such as while (!feof(tosort)) which is always wrong, as explained here: Why is “while ( !feof (file) )” always wrong?

Community
  • 1
  • 1
chqrlie
  • 131,814
  • 10
  • 121
  • 189
  • Added the code on the post. Thanks for mentioning about the stack. I have no clue, but I'll be researching on this. – James Oct 06 '16 at 06:54
  • Why the downvote? the problem was properly diagnosed, extra information was needed to pin it down to a specific part of the source code. – chqrlie Oct 06 '16 at 12:57
  • I didn't realize doing that. Vote button's acting out. Why is `while (!feof(tosort))` wrong? – James Oct 06 '16 at 13:05
  • @James: check this out: http://stackoverflow.com/questions/5431941/why-is-while-feof-file-always-wrong – chqrlie Oct 06 '16 at 13:10