Read 'large' data file in C++

Question

I am trying to reproduce a old R Code I had into C++ in order to gain some speed. This code require the use of preloaded data (6 matrices of dimension 13689 x 126) (each of them weight about 28 MB in the .txt file). It did not cause any problem with R to have this loaded in the memory.

I built a C++ code (I am new to C++) where I try to 'pre-load' these data (is there anyway to pre-load it in the header file btw?).

int i, j;
const int length_grid1 = 13689; 
const int length_grid_pl = 126; 

ifstream in;
//double M1_BETA[length_grid1][length_grid_pl]; // either this, or the other one
std::array<std::array<int, length_grid_pl>, length_grid1> M1_BETA;
in.open("preloaded_object/M1_BETA.txt");
for (i = 0; i < length_grid1; i++) {
    for (j = 0; j < length_grid_pl; j++) { 
        in >> M1_BETA[i][j];
    }
}
in.close();

Both versions, using double or array do not work: I can compile the file (using Intel Composer), but when I launch the .exe, it instantly crashes. This happens with only one of the matrix loaded... And I must load 6 of them.

I don t think it is a problem coming from my code, when I reduce the data dimension it executes itself without any problem.
Notice that I cannot run the program afterwards if everything is not loaded. And reloading it part by part would reduce the pace too much I think (and I switched to C++ in order to gain computation speed..).

Is the program crashing because 28MB is too much? Seems weird because R does not have any problem for example...

Otherwise, can I preload in a way such that it takes less place in memory? I need to have easy access ton any row of the data afterwards (cell by cell is not so important, what I need is really to access specific rows really quickly).

Or is there even another way in C++ to store data, usable at any time?

Thanks in advance.

`std::array, length_grid1> M1_BETA;` is typically instantiated at the stack, that's too large for the stack available in most environments, yes. — πάντα ῥεῖ, Aug 26 '16 at 17:09
It's working with std::vector indeed, thank you! What's the difference with array? — G. Ander, Aug 26 '16 at 18:42

score 0 · Accepted Answer · edited May 23 '17 at 11:53

0

You should use std::vector as std::array is instantiated on the stack. Have a look at http://en.cppreference.com/w/cpp/container/array and std::vector versus std::array in C++.

edited May 23 '17 at 11:53

Community

1
1

answered Aug 26 '16 at 19:00

Giovanni Funchal

8,934
13
61
110

Read 'large' data file in C++

1 Answers1