Opening files over 5 mb and storing them in an array

Question

I want to put each byte in a char array and rewrite the text file removing the first 100,000 characters.

    int fs=0;
    ifstream nm,nm1;

    nm1.open("C:\\Dev-Cpp\\DCS\\Decom\\a.txt");

    if(nm1.is_open())
    {
        nm1.seekg(0, ios::end ); 
        fs = nm1.tellg();

    }
    nm1.close();


    char ss[500000];

    nm.open("C:\\Dev-Cpp\\DCS\\Decom\\a.txt");
    nm.read(ss,fs-1);
    nm.close();

    ofstream om;
    om.open("C:\\Dev-Cpp\\DCS\\Decom\\a.txt");
    for(int i=100000;i<fs-1;i++){
            om >> ss[i];
            }
    om.close();

Problem is i can't set the character array to a 5 million size. I tried using vector also

    vector <char> ss (5000000); 
    int w=0;

    ifstream in2("C:\\Dev-Cpp\\DCS\\Decom\\a.txt", ios::binary);
    unsigned char c2;
    while( in2.read((char *)&c2, 1) )
    {       
    in2 >> ss[w];
    w++;
    }

Over here the size of w is almost half that of fs and a lot of characters are missing.

How to do it ?

Sergey Kalinichenko · Accepted Answer · 2012-01-14T12:24:01.233

2

In most implementations, char ss[5000000] tries allocating on the stack, and the size of the stack is limited as compared to the overall memory size. You can often allocate larger arrays on the heap than on the stack, like this:

char *ss = new char [5000000];
// Use ss as usual
delete[] ss; // Do not forget to delete

Note that if the file size fs is larger than 5000000, you will write past the end of the buffer. You should limit the amount of data that you read:

nm.read(ss,min(5000000,fs-1));

edited Jan 14 '12 at 12:24

answered Jan 14 '12 at 12:01

Sergey Kalinichenko

714,442
84
1,110
1,523

thanks, i've used char *ss = new char [5000000]; as you suggested and delete[] ss; in the end of the program and it works fine now but i have no experience in heaps and stacks and don't know where to place the size_t code ( inside the fstream ? ) – Gambit King Jan 14 '12 at 12:13
@GambitKing Initially you need only cursory understanding of [heap vs. stack](http://stackoverflow.com/questions/79923/what-and-where-are-the-stack-and-heap) that you can gain from five minutes of reading. I was wrong about the `size_t actualSize` - see my edit. – Sergey Kalinichenko Jan 14 '12 at 12:24

score 1 · Answer 2 · answered Jan 14 '12 at 12:12

This part is not correct

while( in2.read((char *)&c2, 1) )
{
    in2 >> ss[w];
    w++;
}

bacause you first try to read one character into c2 and, if that succeeds, read another character into ss[w].

I'm not at all surprised if you lose about half the characters here!

score 0 · Answer 3 · answered Jan 14 '12 at 12:42

The best way to solve your problem is to use the facilities of the standard library. That way, you also don't have to care about buffer overflows.

The following code is untested.

std::fstream file("C:\\Dev-Cpp\\DCS\\Decom\\a.txt", std::ios_base::in);
if (!file)
{
  std::cerr << "could not open file C:\\Dev-Cpp\\DCS\\Decom\\a.txt for reading\n";
  exit(1);
}

std::vector<char> ss; // do *not* give a size here
ss.reserve(5000000);  // *expected* size

// if the file is too large, the capacity will automatically be extended
std::copy(std::istreambuf_iterator<char>(file), std::istreambuf_iterator<char>(),
          std::back_inserter(ss));

file.close();
file.open("C:\\Dev-Cpp\\DCS\\Decom\\a.txt", std::ios_base::out | std::ios_base::trunc);
if (!file)
{
  std::cerr << "could not open C:\\Dev-Cpp\\DCS\\Decom\\a.txt for writing\n";
  exit(1);
}

if (ss.size() > 100000) // only if the file actually contained more than 100000 characters
  std::copy(ss.begin()+100000, ss.end(), std::ostreambuf_iterator<char>(file));

file.close();

Opening files over 5 mb and storing them in an array

3 Answers3