1

Just before using MSVC++ input.getline() to read a very big (3GB) delimited text file, I wanted to optimize the speed, incrementing the size of the input buffer:

    ifstream input("in1.txt");
    input.rdbuf()->pubsetbuf(NULL, 1024 * 1024);

However, when executing the code, the speed did not improve, so I would like to know:

  • What is wrong in the code?
  • Does buffering works with ifstream.getline?
  • What is the size of the default buffering assigned to ifstream?

Regards.

111111
  • 15,686
  • 6
  • 47
  • 62
vizcayno
  • 1,223
  • 3
  • 16
  • 24
  • 2
    possible duplicate of [What is the Fastest Method for High Performance Sequential File I/O in C++?](http://stackoverflow.com/questions/1201261/what-is-the-fastest-method-for-high-performance-sequential-file-i-o-in-c) and [Fastest way to read a text file of strings line by line](http://stackoverflow.com/questions/9356216/fastest-way-to-read-a-text-file-of-strings-line-by-line) – Ben Voigt Feb 23 '12 at 21:18

5 Answers5

3

Did you consider the mmap() system call?

The mmap() function shall establish a mapping between a process' address space and a file, shared memory object, or typed memory object. The format of the call is as follows:

pa=mmap(addr, len, prot, flags, fildes, off);

man page

MapViewOfFile is the windows equivalent.

LPVOID WINAPI MapViewOfFile( __in  HANDLE hFileMappingObject,
__in  DWORD dwDesiredAccess, __in  DWORD dwFileOffsetHigh, __in  DWORD dwFileOffsetLow, __in  SIZE_T dwNumberOfBytesToMap );

yves Baumes
  • 8,836
  • 7
  • 45
  • 74
  • Ives, thanks for your help! although the code for windows looks horrible and very near to C. I can't believe that it's not possible to improve the speed using the STL. – vizcayno Feb 23 '12 at 22:55
0

The thing about buffering is that it works at many levels, you have library (ifstream) level buffering, you have OS level buffering and hardware level buffering. Changing the size of anyone one of those can have a major or non existent impact on performance.

What is true, is that the 'logic' of the program is going to be much faster than that of the IO.

Personally unless the bottle neck is serious I would leave it be.

111111
  • 15,686
  • 6
  • 47
  • 62
  • 1
    "the 'logic' of the program is going to be much faster than that of the IO" This is a common assumption that no one ever tests. They just use it as an excuse not to make I/O faster. As a result of never worrying about I/O performance, it's really really really slow. – Ben Voigt Feb 23 '12 at 21:19
  • @BenVoigt: +1 I agree!. With the rise of 10g and 40g NICs on the market, I/O infact saturates the CPU, "the logic" quickly becomes the bottleneck and dumb/inefficient processing of input within "the logic" is the cause of most slow downs. – Sami Kenjat Nov 11 '12 at 22:30
0

You will get the absolute fastest performance by using CreateFile and ReadFile. Open the file with FILE_FLAGS_SEQUENTIAL_SCAN.

Read with a buffer size that is a power of two. Only benchmarking can determine this number. I have seen it to be 8K once. Another time I found it to be 8M! This varies wildly.

It depends on the size of the CPU cache, on the efficiency of OS read-ahead and on the overhead associated with doing many small writes.

Memory mapping is not the fastest way. It has more overhead because you can't control the block size and the OS needs to fault in all pages.

usr
  • 168,620
  • 35
  • 240
  • 369
  • The accepted answers explains why this is the case. He also did not use FILE_FLAG_SEQUENTIAL_SCAN. – usr Feb 23 '12 at 23:08
  • Yes, I told him that in the comments. But the fact that `MapViewOfFile` performed quite a bit faster than `FILE_FLAG_NO_BUFFERING` shows that memory-mapping is either much more efficient than `ReadFile`, possibly because of not needed to copy everything from the cache to a private page owned by the app, or else memory-mapping does prefetching of its own (which was turned off for `ReadFile` in that question), or maybe some of both. – Ben Voigt Feb 23 '12 at 23:13
  • Yes, memory mapping does prefetching on its own. At least according to the "Windows Internals" book. The Cache Manager can even detect certain access patterns. It switches automatically into SEQUENTIAL or RANDOM_ACCESS mode internally. – usr Feb 24 '12 at 11:45
  • FILE_FLAG_NO_BUFFERING means that you get a disk seek for every block because no prefetching is keeping the drive busy. – usr Feb 24 '12 at 11:46
0

You can try using memory mapped file functionality provided by the OS or, if memory is not an issue, try reading the whole file into memory before processing.

Dmitri Bouianov
  • 528
  • 4
  • 5
-1

I wanted to optimize the speed

Get rid of fstream. iostreams in general are a horrible bottleneck.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720
  • Ben, I was persuaded by google tests that C++ is the fastest language and of course, that includes the STL :-( – vizcayno Feb 23 '12 at 22:46
  • @user255053: I seriously doubt you're using the STL, and even if you are, iostreams have never been part of it. The C++ Standard Library may or may not be well optimized, depending on the implementation and which particular part you're using. – Ben Voigt Feb 23 '12 at 23:00