How to speed-up loading of 15M integers from file stream?

Question

I have an array of precomputed integers, it's fixed size of 15M values. I need to load these values at the program start. Currently it takes up to 2 mins to load, file size is ~130MB. Is it any way to speed-up loading. I'm free to change save process as well.

std::array<int, 15000000> keys;

std::string config = "config.dat";

// how array is saved
std::ofstream out(config.c_str());
std::copy(keys.cbegin(), keys.cend(),
  std::ostream_iterator<int>(out, "\n"));

// load of array
std::ifstream in(config.c_str());
std::copy(std::istream_iterator<int>(in),
  std::istream_iterator<int>(), keys.begin());
in_ranks.close();

Thanks in advance.

SOLVED. Used the approach proposed in accepted answer. Now it takes just a blink.

Thanks all for your insights.

Answer 1

You have two issues regarding the speed of your write and read operations.

First, std::copy cannot do a block copy optimization when writing to an output_iterator because it doesn't have direct access to underlying target.

Second, you're writing the integers out as ascii and not binary, so for each iteration of your write output_iterator is creating an ascii representation of your int and on read it has to parse the text back into integers. I believe this is the brunt of your performance issue.

The raw storage of your array (assuming a 4 byte int) should only be 60MB, but since each character of an integer in ascii is 1 byte any ints with more than 4 characters are going to be larger than the binary storage, hence your 130MB file.

There is not an easy way to solve your speed problem portably (so that the file can be read on different endian or int sized machines) or when using std::copy. The easiest way is to just dump the whole of the array to disk and then read it all back using fstream.write and read, just remember that it's not strictly portable.

To write:

std::fstream out(config.c_str(), ios::out | ios::binary);
out.write( keys.data(), keys.size() * sizeof(int) );

And to read:

std::fstream in(config.c_str(), ios::in | ios::binary);
in.read( keys.data(), keys.size() * sizeof(int) );

----Update----

If you are really concerned about portability you could easily use a portable format (like your initial ascii version) in your distribution artifacts then when the program is first run it could convert that portable format to a locally optimized version for use during subsequent executions.

Something like this perhaps:

std::array<int, 15000000> keys;

// data.txt are the ascii values and data.bin is the binary version
if(!file_exists("data.bin")) {
    std::ifstream in("data.txt");
    std::copy(std::istream_iterator<int>(in),
         std::istream_iterator<int>(), keys.begin());
    in.close();

    std::fstream out("data.bin", ios::out | ios::binary);
    out.write( keys.data(), keys.size() * sizeof(int) );
} else {
    std::fstream in("data.bin", ios::in | ios::binary);
    in.read( keys.data(), keys.size() * sizeof(int) );
}

If you have an install process this preprocessing could also be done at that time...

Answer 2

Attention. Reality check ahead:

Reading integers from a large text file is an IO bound operation unless you're doing something completely wrong (like using C++ streams for this). Loading 15M integers from a text file takes less than 2 seconds on an AMD64@3GHZ when the file is already buffered (and only a bit long if had to be fetched from a sufficiently fast disk). Here's a quick & dirty routine to prove my point (that's why I do not check for all possible errors in the format of the integers, nor close my files at the end, because I exit() anyway).

$ wc nums.txt
 15000000  15000000 156979060 nums.txt

$ head -n 5 nums.txt
730547560
-226810937
607950954
640895092
884005970

$ g++ -O2 read.cc
$ time ./a.out <nums.txt
=>1752547657

real    0m1.781s
user    0m1.651s
sys     0m0.114s

$ cat read.cc 
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <vector>

int main()
{
        char c;
        int num=0;
        int pos=1;
        int line=1;
        std::vector<int> res;
        while(c=getchar(),c!=EOF)
        {
                if (c>='0' && c<='9')
                        num=num*10+c-'0';
                else if (c=='-') 
                        pos=0;
                else if (c=='\n')
                {
                        res.push_back(pos?num:-num);
                        num=0;
                        pos=1;
                        line++;
                }
                else
                {
                        printf("I've got a problem with this file at line %d\n",line);
                        exit(1);
                }
        }
        // make sure the optimizer does not throw vector away, also a check.
        unsigned sum=0;
    for (int i=0;i<res.size();i++) 
    {
    sum=sum+(unsigned)res[i];
    }
    printf("=>%d\n",sum); 
}

UPDATE: and here's my result when read the text file (not binary) using mmap:

$ g++ -O2 mread.cc
$ time ./a.out nums.txt
=>1752547657

real    0m0.559s
user    0m0.478s
sys     0m0.081s

code's on pastebin:

• http://pastebin.com/NgqFa11k

What do I suggest

1-2 seconds is a realistic lower bound for a typical desktop machine for load this data. 2 minutes sounds more like a 60 Mhz micro controller reading from a cheap SD card. So either you have an undetected/unmentioned hardware condition or your implementation of C++ stream is somehow broken or unusable. I suggest to establish a lower bound for this task on your your machine by running my sample code.

Answer 3

if the integers are saved in binary format and you're not concerned with Endian problems, try reading the entire file into memory at once (fread) and cast the pointer to int *

Answer 4

You could precompile the array into a .o file, which wouldn't need to be recompiled unless the data changes.

thedata.hpp:

static const int NUM_ENTRIES = 5;
extern int thedata[NUM_ENTRIES];

thedata.cpp:

#include "thedata.hpp"
int thedata[NUM_ENTRIES] = {
10
,200
,3000
,40000
,500000
};

To compile this:

# make thedata.o

Then your main application would look something like:

#include "thedata.hpp"
using namespace std;
int main() {
  for (int i=0; i<NUM_ENTRIES; i++) {
    cout << thedata[i] << endl;
  }
}

Assuming the data doesn't change often, and that you can process the data to create thedata.cpp, then this is effectively instant loadtime. I don't know if the compiler would choke on such a large literal array though!

Answer 5

Save the file in a binary format.

Write the file by taking a pointer to the start of your int array and convert it to a char pointer. Then write the 15000000*sizeof(int) chars to the file.

And when you read the file, do the same in reverse: read the file as a sequence of chars, take a pointer to the beginning of the sequence, and convert it to an int*.

of course, this assumes that endianness isn't an issue.

For actually reading and writing the file, memory mapping is probably the most sensible approach.

Answer 6

If the numbers never change, preprocess the file into a C++ source and compile it into the application.

If the number can change and thus you have to keep them in separate file that you have to load on startup then avoid doing that number by number using C++ IO streams. C++ IO streams are nice abstraction but there is too much of it for such simple task as loading a bunch of number fast. In my experience, huge part of the run time is spent in parsing the numbers and another in accessing the file char by char.

(Assuming your file is more than single long line.) Read the file line by line using std::getline(), parse numbers out of each line using not streams but std::strtol(). This avoids huge part of the overhead. You can get more speed out of the streams by crafting your own variant of std::getline(), such that reads the input ahead (using istream::read()); standard std::getline() also reads input char by char.

Answer 7

Use a buffer of 1000 (or even 15M, you can modify this size as you please) integers, not integer after integer. Not using a buffer is clearly the problem in my opinion.

Answer 8

If the data in the file is binary and you don't have to worry about endianess, and you're on a system that supports it, use the mmap system call. See this article on IBM's website:

High-performance network programming, Part 2: Speed up processing at both the client and server

Also see this SO post:

When should I use mmap for file access?