C++ Performance writing and reading from disk

Question

Possible Duplicate:
Writing a binary file in C++ very fast

I have a large number of unsigned 32 bit integers in memory (1.5 billion entries). I need to write them to a file and read them back into main memory.

Now, I do it using:

ofstream ofs;
ofs.open(filename);
for (uint64_t i = 0 ; i < 1470000000 ; i++)
ofs << integers << " " ;

and

ifstream ifs;
ifs.open(filename);
for (uint64_t i = 0 ; i < 1470000000 ; i++)
ifs >> integers ;

This takes a few minutes to execute. Can anybody help me, is there any library method to do it in a faster way? Or any suggestion, so I can run a performance test? Can anybody show me some simple C++ code that uses mmap for doing the above (on Linux)?

EDIT: EXAMPLE CASE

#include<iostream>
#include <stdint.h>
#include <cstdio>
#include <cstdlib>
#include <sstream>

using namespace std;

main()
{
      uint32_t* ele = new uint32_t [100] ;
      for(int i = 0; i < 100 ; i++ )
      ele[i] = i ;

      for(int i = 0; i < 100 ; i++ ){
          if(ele[i] < 20)
          continue ;
          else
          // write  ele[i] to file
          ;   
      }

 for(int i = 0; i < 100 ; i++ ){
          if(ele[i] < 20)
          continue ;
          else
          // read  number from file
          // ele[i] = number * 10 ;
          ;   
      }

     std::cin.get();
}

Answer 1

The first thing to do is to determine where the time is going. Formatting and parsing text isn't trivial, and can take some time, but so can the actual writing and reading, given the size of the file. The second thing is to determine how "portable" the data have to be: the fastest solution is almost certainly to mmap (or its Windows equivalent) the array to the file directly, and never read or write. This doesn't provide a portable representation, however, and even upgrading the compiler might make the data unreadable. (Unlikely for 32 bit integers today, but it has happened in the past).

In general, if the time is going to reading and writing, you will want to investigate using mmap. If it is going to formatting and parsing, you will want to investigate some sort of binary format—this could also help reading and writing if it makes the resulting files smaller. The simplest binary format, writing the values using the normal network standard, requires no more than:

void
writeInt( std::ostream& dest, int32_t integer )
{
    dest.put( (integer >> 24) & 0xFF );
    dest.put( (integer >> 16) & 0xFF );
    dest.put( (integer >>  8) & 0xFF );
    dest.put( (integer      ) & 0xFF );
}

int32_t
readInt( std::istream& source )
{
    int32_t results = 0;
    results  = source.get() << 24;
    results |= source.get() << 16;
    results |= source.get() <<  8;
    results |= source.get();
    return results;
}

(Some error checking obviously needs to be added.)

If many of the integers are actually small, you could try some variable length encoding, such as that used in Google Protocol Buffers. If most of your integers are in the range -64...63, this could result in a file only a quarter of the size (which again, will improve the time necessary to read and write).

Answer 2

If you know the size just fwrite/fread an array.

Answer 3

You can likely get better performance by using a bigger buffer for both your input and output streams:

ofstream ofs;
char * obuffer = new char[bufferSize];
ofs.rdbuf ()->pubsetbuf (obuffer, bufferSize);
ofs.open (filename);

ifstream ifs;
char * ibuffer = new char[bufferSize];
ifs.rdbuf ()->pubsetbuf (ibuffer, bufferSize);
ifs.open (filename);

Also ifs >> integers ; is fairly slow way to parse just integers. Try to read lines and then use std::strtol() to parse them. IME, it is measurably faster.

Answer 4

If you just want to copy you can use that for better performance:

std::ifstream  input("input");
std::ofstream  output("ouptut");
output << input.rdbuf();

or maybe setting the buffer size may increase the speed :

char cbuf[buf_size];
ifstream fin;
fin.rdbuf()->pubsetbuf(cbuf,buf_size);

I didn't consider long int issue in my answer because I simply don't know why they should effect stream performance but I hope it helps anyways .