How to read a file in multiple chunks until EOF (C++)

Question

So, here's my problem: I want to make a program that reads chunks of data from a file. Let's say, 1024 bytes per chunk. So I read the first 1024 bytes, perform various operations and then open the next 1024 bytes, without reading the old data. The program should keep reading data untile the EOF is reached.

I'm currently using this code:

std::fstream fin("C:\\file.txt");

vector<char> buffer (1024,0); //reads only the first 1024 bytes
fin.read(&buffer[0], buffer.size());

But how can I read the next 1024 bytes? I was thinking by using a for loop, but I don't really know how. I'm totally a noob in C++, so if anyone can help me out, that would be great. Thanks!

Yes, a loop is the way to go. Also, you should use `fin.gcount()` to figure out who much data was actually read. Also, if you are only reading, you should use `std::ifstream` (and in case you want to read binary data rather than text, you should open the file with `std::ios_base::binary`). — Dietmar Kühl, Jan 03 '14 at 19:50

score 20 · Accepted Answer · edited Jul 12 '20 at 22:48

20

You can do this with a loop:

std::ifstream fin("C:\\file.txt", std::ifstream::binary);
std::vector<char> buffer (1024,0); //reads only the first 1024 bytes

while(!fin.eof()) {
    fin.read(buffer.data(), buffer.size())
    std::streamsize s=fin.gcount();
    ///do with buffer
}

##EDITED

http://en.cppreference.com/w/cpp/io/basic_istream/read

edited Jul 12 '20 at 22:48

orangepips

9,891
6
33
57

answered Jan 03 '14 at 19:52

111111

15,686
6
47
62

@DietmarKühl why no `rdbuf` ? – user2485710 Jan 03 '14 at 19:59
@user2485710: it is too hard to use correctly for beginners. Even experienced users have rarely heart of `std::istream::sentry`. – Dietmar Kühl Jan 03 '14 at 20:01
1

@user2485710 because most people don't want to have to dust of the iostreams doc simply to read a bin file. – 111111 Jan 03 '14 at 20:01
Yes. This one worked perfectly. Thanks to everybody! – jndok Jan 03 '14 at 20:18
1

Does this really work? According to the documentation "read" returns *this. – Vink Aug 12 '16 at 01:11
5

It should be, while (!fin.eof()) { fin.read(buffer.data(), buffer.size()); std::streamsize dataSize = fin.gcount(); } – Vink Aug 12 '16 at 01:38
[Why is iostream::eof inside a loop condition (i.e. `while (!stream.eof())`) considered wrong?](https://stackoverflow.com/questions/5605125/why-is-iostreameof-inside-a-loop-condition-i-e-while-stream-eof-cons) – Some programmer dude Apr 29 '21 at 14:55
4

This is dangerous code - you should continue `while(fin.good())`, because `eof` is only set if the end of file is reached; if there is an error, your loop would continue forever. `good` becomes false if end of file or if `badbit` is set. Also, you should test `if (fin.bad())` after the loop has finished to see if anything went wrong. – Richard Whitehead Nov 26 '21 at 15:33

score 10 · Answer 2 · answered Oct 12 '18 at 11:59

10

Accepted answer doesn't work for me - it doesn't read last partial chunk. This does:

void readFile(std::istream &input, UncompressedHandler &handler) {
    std::vector<char> buffer (1024,0); //reads only 1024 bytes at a time
    while (!input.eof()) {
        input.read(buffer.data(), buffer.size());
        std::streamsize dataSize = input.gcount();
        handler({buffer.begin(), buffer.begin() + dataSize});
    }
}

Here UncompressedHandler accepts std::string, so I use constructor from two iterators.

answered Oct 12 '18 at 11:59

MateuszL

2,751
25
38

Because on last (not full) read, istream goes eof and so evaluates to false and doesn't enter while body – MateuszL Mar 19 '20 at 13:42
Yes I get it now. Thanks. For others, check return value of https://en.cppreference.com/w/cpp/io/basic_istream/read (return the stream itself `*this`), and its bool overloading method https://en.cppreference.com/w/cpp/io/basic_ios/operator_bool . (`1) Returns a null pointer if fail() returns true`). When the stream reaches its end, `fail()` returns `true`. – Rick Mar 19 '20 at 14:00

score 1 · Answer 3 · answered Jan 03 '14 at 20:05

I think you missed up that there is a pointer points to the last place you've visit in the file , so that when you read for the second time you will not start from the first , but from the last point you've visit . Have a look to this code

std::ifstream fin("C:\\file.txt");
char buffer[1024]; //I prefer array more than vector for such implementation

fin.read(buffer,sizeof(buffer));//first read get the first 1024 byte

fin.read(buffer,sizeof(buffer));//second read get the second 1024 byte

so that how you may think about this concept .

score 0 · Answer 4 · answered Jul 23 '21 at 16:20

I think that will work

     #include <stdlib.h>
     #include <stdio.h>
     #include <string.h>
     #include <fstream>
    
    // Buffer size 16 Megabyte (or any number you like)
    size_t buffer_size = 1 << 24; // 20 is 1 Megabyte
    char* buffer = new char[buffer_size];

    std::streampos fsize = 0;
    std::ifstream file("c:\\file.bin", std::ios::binary);

    fsize = file.tellg();
    file.seekg(0, std::ios::end);
    fsize = file.tellg() - fsize;

    int loops = fsize / buffer_size;
    int lastChunk = fsize % buffer_size;

    for (int i = 0; i < loops; i++) {
        file.read(buffer, buffer_size);
        // DO what needs with the buffer
    }

    if (lastChunk > 0) {
        file.read(buffer, lastChunk);
        // DO what needs with the buffer
    }

    delete[] buffer;

How to read a file in multiple chunks until EOF (C++)

4 Answers4

Linked