HUGE .cpp file better than reading from text file?

Question

Is it a legitimate optimisation to simply create a really HUGE source file which initialises a vector with hundreds of thousands of values manually? rather than parsing a text file with the same values into a vector?

Sorry that could probably be worded better. The function that parses the text file in is very slow due to C++'s stream reading being very slow (takes about 6 minutes opposed to about 6 seconds in the C# version.

Would making a massive array initialisation file be a legitimate solution? It doesn't seem elegant, but if it's faster then I suppose it's better?

this is the file reading code:

    //parses the text path vector into the engine
    void Level::PopulatePathVectors(string pathTable)
    {
        // Read the file line by line.
        ifstream myFile(pathTable);

            for (unsigned int i = 0; i < nodes.size(); i++)
            {
                pathLookupVectors.push_back(vector<vector<int>>());

                for (unsigned int j = 0; j < nodes.size(); j++)
                {
                    string line;

                    if (getline(myFile, line)) //enter if a line is read successfully
                    {
                        stringstream ss(line);
                        istream_iterator<int> begin(ss), end;
                        pathLookupVectors[i].push_back(vector<int>(begin, end));
                    }
                }
            }
        myFile.close();
    }

sample line from the text file (in which there are about half a million lines of similar format but varying length.

0 5 3 12 65 87 n

Answer 1

First, make sure you're compiling with the highest optimization level available, then please add the following lines marked below, then test again. I doubt this will fix the problem, but it may help. Hard to say until I see the results.

//parses the text path vector into the engine
void Level::PopulatePathVectors(string pathTable)
{
    // Read the file line by line.
    ifstream myFile(pathTable);

    pathLookupVectors.reserve(nodes.size()); // HERE
    for (unsigned int i = 0; i < nodes.size(); i++)
    {
        pathLookupVectors.push_back(vector<vector<int> >(nodes.size()));
        pathLookupVectors[i].reserve(nodes.size());  // HERE

        for (unsigned int j = 0; j < nodes.size(); j++)
        {
            string line;

            if (getline(myFile, line)) //enter if a line is read successfully
            {
                stringstream ss(line);
                istream_iterator<int> begin(ss), end;
                pathLookupVectors[i].push_back(vector<int>(begin, end));
            }
         }
     }
     myFile.close();
}

Answer 2

6 minutes vs 6 seconds!! must be something wrong with your C++ code. Optimize it using good old methods before you revert to such an extreme "optimization" mentioned in your post.

Also know that reading from file would allow you to change the vector contents without changing the source code. If you do it the way you mention it, you'll have to re-code, compile n link all over again.

Answer 3

Depending if the data changes. If the data can/needs to be changed (after compiletime) than the only option is to load it from textfile. If not, well I don't see any harm to compile it.

Answer 4

I was able to get the following result with Boost.Spirit 2.5:

$ time ./test input

real    0m6.759s
user    0m6.670s
sys     0m0.090s

'input' is a file containing 500,000 lines containing 10 random integers between 0 and 65535 each.

Here's the code:

#include <vector>

#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/classic_file_iterator.hpp>

using namespace std;
namespace spirit = boost::spirit;
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;

typedef vector<int> ragged_matrix_row_type;
typedef vector<ragged_matrix_row_type> ragged_matrix_type;


template <class Iterator>
struct ragged_matrix_grammar : qi::grammar<Iterator, ragged_matrix_type()> {

  ragged_matrix_grammar() : ragged_matrix_grammar::base_type(ragged_matrix_) {

    ragged_matrix_ %= ragged_matrix_row_ % qi::eol;
    ragged_matrix_row_ %= qi::int_ % ascii::space;

  }

  qi::rule<Iterator, ragged_matrix_type()> ragged_matrix_;
  qi::rule<Iterator, ragged_matrix_row_type()> ragged_matrix_row_;

};

int main(int argc, char** argv){

  typedef spirit::classic::file_iterator<> ragged_matrix_file_iterator;

  ragged_matrix_type result;
  ragged_matrix_grammar<ragged_matrix_file_iterator> my_grammar;
  ragged_matrix_file_iterator input_it(argv[1]);

  qi::parse(input_it, input_it.make_end(), my_grammar, result);

  return 0;

}

At this point, result contains the ragged matrix, which can be confirmed by printing its contents. In my case the 'ragged matrix' isn't so ragged-it's a 500000 x 10 rectangle-but it won't matter because I'm pretty sure the grammar is correct. I got even better results when I read the entire file into memory before parsing (~4 sec), but the code for that is longer and it's generally undesirable to copy large files into memory in their entirety.

Note: my test machine has an SSD, so I don't know if you'll get the same numbers I did (unless your test machine has an SSD as well).

HTH!

Answer 5

I wouldn't consider compiling static data into your application to be bad practice. If there is little conceivable need to change your data without a recompilation, parsing the file at compile time not only improves runtime performance (since your data have been pre-parsed by the compiler and are in a usable format at runtime), but also reduces risks (like the data file not being found at runtime or any other parse errors).

Make sure that users won't have need to change the data (or have the means to recompile the program), document your motivation and you should be absolutely fine.

That said, you could make the iostream version a lot faster if necessary.

Answer 6

using a huge array in a C++ file is a totally allowed option, depending on the case.

You must consider if the data will change and how often.

If you put it in a C++ file, that means that you will have to recompile your program each time the data change (and distribute it to your customers each time !) So that wouldn't be a good solution if you have to distribute the program to other people.

Now if a compilation is allowed for every data change, then you can have the best of two worlds : just use a small script (for example in python or perl) which will take your .txt and generate a C++ file, so the file parsing will only have to be done one time for each data change. You can even integrate this step in your build process with automatic dependency management.

Good luck !

Answer 7

Don't use the std input stream, it's extremely slow. There are better alternatives.

Since people decided to downvote my answer because they are too lazy to use google, here:

http://accu.org/index.php/journals/1539