How to read a big data in C?

Question

I practice with file handling in C. I need to read a file which recorded about 9000000 data. It works when I read the first 9000 data. However, it always crashes when I set NUM_DATA = 9000000. How can I solve this problem?

#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdlib.h>

#define MAX_LENGTH 10
#define NUM_DATA 9000000

int main(int argc, char **argv){

    char **signal = malloc( NUM_DATA * MAX_LENGTH * sizeof(char *) );
    char **svid = malloc( NUM_DATA * MAX_LENGTH * sizeof(char *) );

    char GetString[110];
    int num;
    double tw[NUM_DATA], ip[NUM_DATA], qp[NUM_DATA]
    char a[20], b[20], c[20], d[20], e[20], f[20], g[20], h[20], z[20], x[20], w[20];

    FILE *fIn;

    fIn = fopen(argv[1], "r");

    // read header
    fgets(GetString, 109, fIn);
    fgets(GetString, 109, fIn);

    // read data
    num = 0;
    while (!feof(fIn)){

        fscanf(fIn, "%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%f",
               a, b, c, d, e, f, g, h, z, x, w);

        tw[num] = atof(a);
        ip[num] = atof(z);
        qp[num] = atof(x);
        signal[num] = malloc(MAX_LENGTH * sizeof(char *));
        strcpy(signal[num], f);
        strcpy(svid[num], h);
        num++;
    }
    fclose(fIn);
}

The format of the file:

353700.000,1352,0.020000,0.000000,1,ISR1,Main,E11,173,14,56.102000 353700.000,1353,0.020000,0.000000,1,ISR1,Main,E11,156,7,14.367000 353700.000,1354,0.020000,0.000000,1,ISR1,Main,E11,161,14,40.136000 353700.000,1355,0.020000,0.000000,1,ISR1,Main,E11,316,23,31.326000 353700.000,1356,0.020000,0.000000,1,ISR1,Main,E11,152,-1,3.806000 353700.000,1357,0.020000,0.000000,2,ISR1,Main,G14,-101,10,51.833000 353700.000,1358,0.020000,0.000000,3,ISR1,Main,G24,434,-3,56.414000 353700.000,1359,0.020000,0.000000,3,ISR1,Main,G24,171,-4,17.890000 353700.000,1360,0.020000,0.000000,3,ISR1,Main,G24,474,6,52.542000

......

Do you need to have all the data in-memory at the same time, or can you aggregate the data in chunks? — Matthew, Nov 24 '21 at 20:44
It is crashing because you allocate your arrays on stack. Which has a limited capacity. You better go with dynamic allocation (`malloc`/`free` and friends). This can fail too of course, but in more controllable way. — Eugene Sh., Nov 24 '21 at 20:44
also see [Why is “while ( !feof (file) )” always wrong?](https://stackoverflow.com/questions/5431941/why-is-while-feof-file-always-wrong) — yano, Nov 24 '21 at 20:46
@Cheatah like Fourier Transform or other time-series analysis. — Chun Yen Huang, Nov 25 '21 at 07:34

score 5 · Accepted Answer · answered Nov 24 '21 at 20:44

5

You're overflowing the stack with local arrays of that size. Allocate them dynamically instead.

double *tw = malloc(NUM_DATA * sizeof(double));
double *ip = malloc(NUM_DATA * sizeof(double));
double *qp = malloc(NUM_DATA * sizeof(double));

answered Nov 24 '21 at 20:44

dbush

205,898
23
218
273

Thank you for your answer. I have allocated the three variables and svid[num] in dynamic allocation, and it works. – Chun Yen Huang Nov 25 '21 at 07:47

How to read a big data in C?

1 Answers1