Problems reading a file and passing into a large multi dimensional array

Question

I am trying to read a large data file and passing the data as a 9-dimensional array B[37][24][5][5][8][19][6][19][14]. I will have to use this array later in the code. But I am getting the Segmentation fault error.

I have also tried to define B as a pointer, instead of an array and used the command "B = (double *)malloc(5385542400 * sizeof(double));". But it didn't help.

Here is the code where I defined B as an array.

#include <stdio.h>
#include <stdlib.h>
#include <gsl/gsl_rng.h>
#include <gsl/gsl_randist.h>
#include <math.h>
#include <string.h>



/* If filename given, write to file; for empty filename write to 
screen */
char MYFILE[]="chisq_NHtest_NOvA_nu_app_non-uni_alpha_phi10_10sys.dat";
int main()
{
FILE *outfile = NULL; 

  outfile = fopen(MYFILE, "w");
  if (outfile == NULL)
  {
printf("Error opening output file.\n");
return -1;
  }

char MYFILE1[]="nova_app_nu_data.dat";

FILE *file1 = NULL;
file1 = fopen(MYFILE1, "r");
  if (file1 == NULL)
  {
    printf("Error reading input file1.\n");
    return -1;
  }
int i;
static double A[6];
double ret1;
 for (i=0; i<6; i++)
            {

             ret1= fscanf(file1,"%lf ",&A[i]);

    if (ret1 == EOF)
    {
    break;
    }
}


char MYFILE2[]="events_vs_E+test_NH_NOvA_nu_app_non-uni_phi10_alpha00_alpha10_alpha11.dat";

FILE *file2 = NULL;
file2 = fopen(MYFILE2, "r");
  if (file2 == NULL)
  {
    printf("Error reading input file2.\n");
    return -1;
  }
int k,l,m,n,o,p,q,r,s;
double C;
static double B[37][24][5][5][8][19][6][19][14];

double ret2;
 for (k=0; k<37; k++)
            {
 for (l=0; l<24; l++)
        {
for (m=0;m<5;m++)
         {
for (n=0;n<5;n++)
         {
for (p=0; p<8; p++)
        {
for (q=0; q<19; q++)
        {
for (r=0; r<6; r++)
        {
for (s=0; s<19; s++)
        {
for (o=0;o<14;o++)
{
          fscanf(file2,"%lf ",&ret2);
B[k][l][m][n][p][q][r][s][o]=ret2;
if (ret2 == EOF)
{
break;
}
}
}
}
}
}
}
}
}
}

exit(0);
}

If I change the end value of s loop from 19 to 5, the code is running properly.

Maybe you should rethink your design if you have 9 dimensions in your array. — Neijwiert, Jul 02 '19 at 11:04
Do your system have over ***forty*** gigs of memory available as a contiguous chunk? — Some programmer dude, Jul 02 '19 at 11:09
You furthermore need to go back to your books, tutorials or class-notes, because there's some misunderstanding in your knowledge about how [`fscanf` (and related functions)](https://en.cppreference.com/w/c/io/fscanf) works. — Some programmer dude, Jul 02 '19 at 11:15
I don't mean to be rude, but the problem is that you simply don't know what you are doing. Assuming double is 8 bytes, your array takes up 43Gb of stack space. If you really have files that large, you have to read them in smaller chunks. — Lundin, Jul 02 '19 at 11:20
OT: Just out of curiosity, what are you storing in that 9-dimensional array? Quantum states? — Bob__, Jul 02 '19 at 11:29
@Bob__ I am storing some experimental event numbers as a function of different quantum mechanical parameters. — Ushak, Jul 02 '19 at 13:06
@Lundin... I think you don't understand the problems. I can't afford to read the files in small chunks. These numbers have some physical meaning and I need to use them later in the code and reading them in smaller chunks doesn't serve the purpose. — Ushak, Jul 02 '19 at 13:10
@Ushak Oh? What kind of super computer are you using then, that has a stack size of 43 Gb? I don't think the gsl library has been ported to secret, exotic super computers, so you should perhaps aim to write code for a PC instead. In which case the linked duplicate answers the question. — Lundin, Jul 02 '19 at 13:15
@Lundin That's the problem.. I don't have a super computer. Probably, I have to find some server to run the code, because if I break the file in small chunks the data simply lose their physical implications. — Ushak, Jul 02 '19 at 13:17
@Ushak You'll have to invent an OS with 43Gb stack space. The seg fault is caused by a stack overflow when you go beyond the stack space given to your process. Similarly, your average PC will refuse to allocate 43Gb heap because there isn't that much physical RAM to be had. So you have to rethink your whole program design and use some manner of random access file instead. — Lundin, Jul 02 '19 at 13:35
I'm wondering if it would be possible for you to post a *small* sample of that input file (how big is it, BTW). The (very) little I was supposed to know about QM expired years ago, but that data structure still seems odd to me. — Bob__, Jul 02 '19 at 13:35
@Bob__ a small sample of the file. The file is 35.1 GB. -180 0.4 0.07812 0.00208 0.93 0 0.95 -180 4.78494 15.0508 10.4624 3.53272 1.96088 1.14648 -180 0.4 0.07812 0.00208 0.93 0 0.95 -160 4.78494 15.0508 10.4624 3.53272 1.96088 1.14648 — Ushak, Jul 02 '19 at 13:56
@Lundin Can you kindly help me with random accessing the file? I am not familiar with this kind of programming. — Ushak, Jul 02 '19 at 13:58

score -1 · Answer 1 · answered Jul 02 '19 at 11:19

The break statement that you have written will only exit the innermost for loop.

To exit the other loops, you need to write if conditions to check if the break has been triggered. But, since you have 9 nested for loops, you may be better off using a goto statement.

double ret2;
int ret3;
for (k=0; k<37; k++)
{
  for (l=0; l<24; l++)
  {
    for (m=0;m<5;m++)
    { 
      for (n=0;n<5;n++)
      {
        for (p=0; p<8; p++)
        {
          for (q=0; q<19; q++)
          {
            for (r=0; r<6; r++)
            {
              for (s=0; s<19; s++)
              {
                for (o=0;o<14;o++)
                {
                   ret3 = fscanf(file2,"%lf ",&ret2);
                   B[k][l][m][n][p][q][r][s][o]=ret2;
                   if (ret3 != 1)
                   {
                      goto endloop;
                   }
                }
              }
            }
          }
        }
      }
    } 
  }
}
:endloop

exit(0);

You should also look into your design as to WHY you need a 9 dimensional array and try to simplify this.

That still wouldn't explain the seg-fault. probably because of what @Someprogrammerdude said regarding the size required for that array. — Neijwiert, Jul 02 '19 at 11:21

Problems reading a file and passing into a large multi dimensional array

1 Answers1