C++ ifstream.getline() significantly slower than Java's BufferedReader.readLine()?

Question

I'm in the process of rewriting one of my Android applications to take advantage of the NDK and one of the first things it has to do every time is open a 1.5MB text file (approximately 150k lines) and put every line in a data structure. When I did this operation using Java's BufferedReader.readLine(), reading the file from the SD card takes ~2.5 seconds. Here's the code I used for this:

try {
    BufferedReader br = new BufferedReader(new FileReader("/sdcard/testfile.txt"));
    String thisLine;
    while ((thisLine = br.readLine()) != null) {
        Log.d(TAG, thisLine);
    }
} catch (IOException e) {
    //Log error
}

Using C++ with ifstream takes MUCH longer...around 3 minutes for the same file. Here's the code I used in C++:

char buffer[256];
ifstream ifs;
ifs.open("/sdcard/testfile.txt", ifstream::in);
if (ifs.is_open()) {
    while (!ifs.eof()) {
        ifs.getline (buffer,100);
        LOGD(buffer);
    }
}

I'm pretty rusty on C++, but can't think of any logical explanation for the increased read time. For a while I thought it might be the LOGD function, but I tried taking that out altogether and the read time wasn't really helped much at all. Does anyone have any ideas on what the issue could be? Is there any faster way to read a file line by line in C++? Thanks.

Yep, It's compiled with -O2 (tried -O3 as well). Debugging is turned off. — Eric, Jul 25 '11 at 18:48
I notice you tagged this android, I'm no android expert but is it possible the c++ code is calling back to "java" code to do the I/O operations? — jcoder, Jul 25 '11 at 19:15
@sth: Sorry, I tried posting this from memory. It's been fixed in the OP. — Eric, Jul 25 '11 at 19:48
@JohnB: It's definitely not calling back to Java for anything. — Eric, Jul 25 '11 at 19:48
Are you sure it's the actual reading from file that's slow and not `LOGD`? — someguy, Jul 25 '11 at 20:08
@someguy: Please read the post. "For a while I thought it might be the LOGD function, but I tried taking that out altogether and the read time wasn't really helped much at all." — Fred Larson, Jul 25 '11 at 20:12
@someguy: It happens. I've failed reading comprehension several times on SO. 8v) — Fred Larson, Jul 25 '11 at 20:42

score 14 · Accepted Answer · answered Jul 25 '11 at 19:20

14

One thought is the stdio synchronization might be slowing you down. That can be turned off. I don't know if that would account for all of the difference, but you could try. Also, you're not using eof() correctly. Finally, I'd use the std::string version of getline()

std::ios::sync_with_stdio(false);
ifstream ifs("/sdcard/testfile.txt");
std::string line;
while (getline(ifs, line))
{
    LOGD(line);
}

I haven't tested this code, but you can try it and see if it makes a difference.

answered Jul 25 '11 at 19:20

Fred Larson

60,987
18
112
174

Thanks for the reply, but this doesn't seem to have made a difference. – Eric Jul 25 '11 at 23:51
Spoke too soon...this fixed it. It now reads the file in roughly the same time as the Java version. Thanks a lot! – Eric Jul 26 '11 at 00:03
Great, I just tried `ios::sync_with_stdio(false)` in my program and it sped up getline 7 times, thanks for great tip – NoxArt Apr 11 '13 at 16:05

score 4 · Answer 2 · answered Jul 25 '11 at 19:21

4

Is it possible that the stream is unbuffered, and it's doing an SD access for each byte of data? To provide a buffer, do the following (size as you feel appropriate).

ifstream ifs;
char stream_buffer[4096];
ifs.rdbuf()->pubsetbuf(stream_buffer, sizeof(stream_buffer) );
ifs.open(argv[1]);

answered Jul 25 '11 at 19:21

Dave S

20,507
3
48
68

This sounded really promising to me, but doesn't seem to have helped. Thanks for replying anyway! – Eric Jul 25 '11 at 23:49

Tom Kerr · Answer 3 · 2011-07-26T14:26:14.517

C++ does not buffer streams for you (edit: they won't by default, see Dave Smith's solution). I will tell you that your code will be slow on a normal platter based disk. I don't have a lot of experience with android, fwiw.

I generally use something like this:

struct buffered_reader {
    buffered_reader(std::istream &data_) : data(data_), done(false) {}
    bool next(std::string &line) {
        if (!lines.size()) {
            if (done)
                return false;
            std::string line;
            for (size_t i = 0; i < 500; i++) {
                std::getline(data, line);
                if (data.eof()) {
                    done = true;
                    break;
                }
                lines.push_back(line);
            }
        }
        line = lines.front();
        lines.pop_front();
        return true;
    }
    std::istream &data;
    bool done;

    std::deque<std::string> lines;
};

TEST(blah) {
    std::stringstream ss;
    ss << "a" << std::endl;
    ss << "a" << std::endl;
    ss << "a" << std::endl;
    ss << "a" << std::endl;

    buffered_reader reader(ss);
    std::string line;
    while(reader.next(line)) {
        std::cout << line << std::endl;
    }
}

This isn't in production anywhere, so no warranties beyond the testing you see here ;)

You say C++ doesn't buffer streams, but the documentation seems to indicate otherwise. It seems that all ifstream objects have an associated filebuf object that they use to buffer all input. Am I understanding this wrong? Thanks for the reply. Source: [cplusplus.com](http://www.cplusplus.com/reference/iostream/ifstream/) — Eric, Jul 26 '11 at 01:33
@Eric Ah, in my head that read "C++ does not buffer streams for you **by default**" and the last bit was lost. Using Dave Smith's solution is perfectly fine, I tend to use/want line based buffering is all. I suspect amortized time is equivalent. — Tom Kerr, Jul 26 '11 at 14:25

C++ ifstream.getline() significantly slower than Java's BufferedReader.readLine()?

3 Answers3

Linked