5

I'm working on a C++ program (C++ 98). It reads a text file with lots of lines (10000 lines). These are tab separated values and then I parse it into Vector of Vector objects. However It seems to work for some files (Smaller) but One of my files gives me the following error (this file has 10000 lines and it's 90MB). I'm guessing this is a memory related issue? Can you please help me?

Error

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Abort

Code

void AppManager::go(string customerFile) {

    vector<vector<string> > vals = fileReader(customerFile);

    for (unsigned int i = 0; i < vals.size();i++){

        cout << "New One\n\n";

        for (unsigned int j = 0; j < vals[i].size(); j++){

            cout << vals[i][j] << endl;
        }

        cout << "End New One\n\n";
    }
}

vector<vector<string> > AppManager::fileReader(string fileName) {

    string line;
    vector<vector<string> > values;

    ifstream inputFile(fileName.c_str());

    if (inputFile.is_open()) {

        while (getline(inputFile,line)) {

            std::istringstream iss(line);
            std::string val;
            vector<string> tmp;

            while(std::getline(iss, val, '\t')) {

                tmp.push_back(val);
            }

            values.push_back(tmp);
        }

        inputFile.close();
    }
    else {

        throw string("Error reading the file '" + fileName + "'");
    }

    return values;
}
Achintha Gunasekara
  • 1,165
  • 1
  • 15
  • 29
  • 1
    Do you have a stack trace that would show where the `bad_alloc` is thrown? (Also, how much free RAM do you have?) – us2012 Oct 08 '13 at 00:20
  • I'm using a server at my university, which I have no control over. It's ruining Solaris and my application is expected to run on it. Data set was provided by them – Achintha Gunasekara Oct 08 '13 at 00:25
  • How do I get a stack trace? Sorry I'm new to C++ – Achintha Gunasekara Oct 08 '13 at 00:25
  • To get a stack trace, run your program under the debugger. When the exception is thrown, the debugger will give you back the backtrace. You may have `gdb` on your platform, Solaris also has `dbx`. Generally you use a debugger by giving the executable to be debugged on the command line, then issuing the `run` command. Alas, the manual is your friend. – Kuba hasn't forgotten Monica Oct 08 '13 at 02:40

1 Answers1

7

There's nothing wrong with your code, you're simply running on a platform likely with small memory limits, likely an old compiler and likely an old C++ library. It all conspires against you. You'll have to micro-optimize :(

Here's what you can do, starting with lowest hanging fruit first:

  1. Do a dry run through the file, just counting the lines. Then values.resize(numberOfLines) , seek to the beginning and only then read the values. Of course you won't be using values.push_back anymore, merely values[lineNumber] = tmp. Resizing the values vector as you add to it may more than double the amount of memory your process needs on a temporary basis.

  2. At the end of the line, do tmp.resize(tmp.size() - it'll shrink the vector to just fit the data.

  3. You can reduce overheads in the existing code by storing all the values in one vector.

    1. If each line has a different number of elements, but you access them sequentially later on, you can store an empty string as an internal delimiter, it may have lower overhead than the vector.

    2. If each line has same number of values, then splitting them by lines adds unnecessary overhead - you know the index of the first value in each line, it's simply lineNumber * valuesPerLine, where first line has number 0.

  4. Memory-map the file. Store the beginning and end of each word in a structure element of a vector, perhaps with a line number as well if you need it split up in lines.

Kuba hasn't forgotten Monica
  • 95,931
  • 16
  • 151
  • 313