C++ fast file reading line by line

Question

I want to create a simple .OBJ parser. Code below works, but is slow. Im testing it with a 26MB file and it takes about 22 seconds to parse it. If i commented out the lines seen in the code, the time went to 17 seconds. It takes 17 seconds to iterate the file line by line and to extract data. Is there any way i can make it faster? I also tried reading the file into memory and then parsing it. It became slower.

while (std::getline(file, line)) {
        if (line[0] == 'v') {
            std::istringstream iss(line);
            std::string type;
            GLfloat x, y, z;
            iss >> type >> x >> y >> z;
            if (type.compare("v") == 0) {
                //vertices.push_back(x);
                //vertices.push_back(y);
                //vertices.push_back(z);
            }
            else if (type.compare("vn") == 0) {
                //normals.push_back(x);
                //normals.push_back(y);
                //normals.push_back(z);
            }
            else if (type.compare("vt") == 0) {
                //UVs.push_back(x);
                //UVs.push_back(y);
            }
        }
        else if (line[0] == 'f') {
            unsigned int size = out_vertices.size() / 3;
            unsigned int vertexIndex[3], uvIndex[3], normalIndex[3];
            sscanf_s(line.c_str(),
                "f %d/%d/%d %d/%d/%d %d/%d/%d",
                &vertexIndex[0], &uvIndex[0], &normalIndex[0],
                &vertexIndex[1], &uvIndex[1], &normalIndex[1],
                &vertexIndex[2], &uvIndex[2], &normalIndex[2]);

            /*GLfloat* vertex = &vertices[(vertexIndex[0] - 1) * 3];
            out_vertices.insert(out_vertices.end(), vertex, vertex + 3);

            vertex = &vertices[(vertexIndex[1] - 1) * 3];
            out_vertices.insert(out_vertices.end(), vertex, vertex + 3);

            vertex = &vertices[(vertexIndex[2] - 1) * 3];
            out_vertices.insert(out_vertices.end(), vertex, vertex + 3);

            GLfloat* uv = &UVs[(uvIndex[0] - 1) * 2];
            out_UVs.insert(out_UVs.end(), uv, uv + 2);

            uv = &UVs[(uvIndex[1] - 1) * 2];
            out_UVs.insert(out_UVs.end(), uv, uv + 2);

            uv = &UVs[(uvIndex[2] - 1) * 2];
            out_UVs.insert(out_UVs.end(), uv, uv + 2);

            GLfloat* normal = &normals[(normalIndex[0] - 1) * 3];
            out_normals.insert(out_normals.end(), normal, normal + 3);

            normal = &normals[(normalIndex[1] - 1) * 3];
            out_normals.insert(out_normals.end(), normal, normal + 3);

            normal = &normals[(normalIndex[2] - 1) * 3];
            out_normals.insert(out_normals.end(), normal, normal + 3);


            out_indices.push_back(size);
            out_indices.push_back(size + 1);
            out_indices.push_back(size + 2);*/
        }
    }

Could you upload somewhere your big file? I'd like to see what happens. 1st rule of speed optimization - profile slow. Have you done any profiling to see where do you spend the most of time? And if the time spent ion getline, I'd assure that file reads are caching forward or switch to memory mapped file. — LiMar, Jun 16 '20 at 15:06
You'll see a list of suggestions here https://stackoverflow.com/questions/5166263/how-to-get-iostream-to-perform-better . If you even better performance, you'll need to go into Win32 API handling or use the Boost library. — Guy Keogh, Jun 16 '20 at 15:11
Does this answer your question? [How to get IOStream to perform better?](https://stackoverflow.com/questions/5166263/how-to-get-iostream-to-perform-better) — Guy Keogh, Jun 16 '20 at 15:12
@LiMar http://www.cadnav.com/3d-models/model-50432.html this is the model. — Marko Taht, Jun 16 '20 at 15:18
What generated the OBJ file? Compilers can generate "obj" files and each have proprietary formats. — Thomas Matthews, Jun 16 '20 at 15:33
Your biggest bottleneck is reading from the file. Reading from the file requires start up time, time to find the file (seek time), then time to read the data from the file. The startup and seek occur for every transaction. You can improve the read speed by increasing the amount of data per transaction. For example, one read transaction of 2048 bytes is more efficient than 2048 requests to read 1 byte. Also search the internet for "memory mapped files" and see if your OS supports them. — Thomas Matthews, Jun 16 '20 at 15:37
Another bottleneck is the (re)allocation of memory for the vector. You can reduce the number of reallocations by allocating a large size for the vector at definition (look at the constructors for `std::vector`). — Thomas Matthews, Jun 16 '20 at 15:39
@ThomasMatthews it is a 3D model. Wavefront OBJ format. I simplified the original by importing exporting it to blender and makeing all the faces a triangle. — Marko Taht, Jun 16 '20 at 15:41
Probably next in line is the conversion of formatted text or parsing. One way around this is to have the file created with fixed length records. Fixed length allows you to jump to locations in the string without parsing. — Thomas Matthews, Jun 16 '20 at 15:41

score 1 · Answer 1 · answered Jun 16 '20 at 15:15

1

Splitting the file into smaller chunks and reading from those chunks might be able to speed up your processing.

Here is what I think might be able to help you:

File Split Into Threads

Similar topic on Stackoverflow

Thanks

answered Jun 16 '20 at 15:15

Somdeb Mukherjee

158
5

Chunking migh help. ALthough threading migh be difficult, as the order of lines is important. – Marko Taht Jun 16 '20 at 15:20
Okay. What if you add a line number to the large file. Do a file split and then a multi-threaded read/processing and then using those line numbers put the data back in order Does it help ? – Somdeb Mukherjee Jun 16 '20 at 16:08

score 1 · Answer 2 · answered Jun 16 '20 at 16:13

1

First, and most important. Rebuild your software in "Release"-mode, not in "Debug"-mode. Or, switch on optimization options on manually.

Then, after opening the file, add the following statements, after the stream is open:

constexpr size_t SizeOfIOStreamBuffer = 100'000;
static char ioBuffer[SizeOfIOStreamBuffer];
file.rdbuf()->pubsetbuf(ioBuffer, SizeOfIOStreamBuffer);

Then, for your std::vectors, please call their reserve(50000U) function.

I expect a execution time to be 1 second. (On my machine with 50MB file it was)

Please try and feed back.

answered Jun 16 '20 at 16:13

A M

14,694
5
19
44

Tested just release mode. and got 1 second. reserver and buffersize didnt make any difference. – Marko Taht Jun 16 '20 at 16:42
having just worked on an issue exactly like this, I suggest trying a bunch of different buffer sizes. There is always a sweet spot. – EvilTeach Jun 17 '20 at 13:18

score 0 · Answer 3 · answered Jun 17 '20 at 13:12

This how I did to optimize this code...

Environment : MacBook Pro, macos, Xcode for IDE. Built-in Instruments for profiling. Target File: downloaded cadnav-200615151218.rar from link you provided. Extracted APC.obj(~17M) and work on it.

Step 1 - reproduce the problem Add missing bits to the code:

typedef float GLfloat;
void reader()
{
    std::ifstream file("/Users/mlifshits/Downloads/APC.obj");

Comment out unsigned int si_ze = out_vertices.size() / 3; since it is not used anywhere and the type of out_vertices unknown.

Profile:

took 537ms, time hogs: operator>>(&float), getline, scanf

Step 2 - attack first time hog which is >>(&float)

Use sscanf() in place of iss

Profile:

took 371ms (improvement!), hogs: scanf and getline

Steps 3 and 4 - attack the hogs!

looked up online about scanf and found that it's slow in reading floats. Replaced with "naive loop" from https://tinodidriksen.com/2011/05/cpp-convert-string-to-double-speed/

since getline hog is so hungry because of multiple allocations (seen in profiler), replaced it with C code using fopen() and fgets()

Profile: took 166ms (vs 546 initially!), hogs still scanf and fgets.

Optimize more? - possible very much. I stop here.

Code: #include #include #include #include #include #include

typedef float GLfloat;

//https://tinodidriksen.com/uploads/code/cpp/speed-string-to-double.cpp
float naive_atof(const char *p) {
    float r = 0.0;
    bool neg = false;
    if (*p == '-') {
        neg = true;
        ++p;
    }
    while (*p >= '0' && *p <= '9') {
        r = (r*10.0) + (*p - '0');
        ++p;
    }
    if (*p == '.') {
        float f = 0.0;
        int n = 0;
        ++p;
        while (*p >= '0' && *p <= '9') {
            f = (f*10.0) + (*p - '0');
            ++p;
            ++n;
        }
        r += f / std::pow(10.0, n);
    }
    if (neg) {
        r = -r;
    }
    return r;
}

void reader()
{
    char line[256] = {0};
    std::string type;
    char xyz[3][100] = {0};
    FILE*fp = fopen("/Users/limar/Downloads/APC.obj", "r");

    while (fgets(line, sizeof(line), fp)) {
        if (line[0] == 'v') {
            GLfloat x, y, z;
            sscanf(line,"%s %s %s %s", (char*)type.c_str(), xyz[0], xyz[1], xyz[2]);
            x = naive_atof(xyz[0]);
            y = naive_atof(xyz[1]);
            z = naive_atof(xyz[2]);
            if (type.compare("v") == 0) {
                //vertices.push_back(x);
                //vertices.push_back(y);
                //vertices.push_back(z);
            }
            else if (type.compare("vn") == 0) {
                //normals.push_back(x);
                //normals.push_back(y);
                //normals.push_back(z);
            }
            else if (type.compare("vt") == 0) {
                //UVs.push_back(x);
                //UVs.push_back(y);
            }
        }
        else if (line[0] == 'f') {
            //unsigned int size = out_vertices.size() / 3;
            unsigned int vertexIndex[3], uvIndex[3], normalIndex[3];
            sscanf(line,
                "f %d/%d/%d %d/%d/%d %d/%d/%d",
                &vertexIndex[0], &uvIndex[0], &normalIndex[0],
                &vertexIndex[1], &uvIndex[1], &normalIndex[1],
                &vertexIndex[2], &uvIndex[2], &normalIndex[2]);

            /*GLfloat* vertex = &vertices[(vertexIndex[0] - 1) * 3];
            out_vertices.insert(out_vertices.end(), vertex, vertex + 3);

            vertex = &vertices[(vertexIndex[1] - 1) * 3];
            out_vertices.insert(out_vertices.end(), vertex, vertex + 3);

            vertex = &vertices[(vertexIndex[2] - 1) * 3];
            out_vertices.insert(out_vertices.end(), vertex, vertex + 3);

            GLfloat* uv = &UVs[(uvIndex[0] - 1) * 2];
            out_UVs.insert(out_UVs.end(), uv, uv + 2);

            uv = &UVs[(uvIndex[1] - 1) * 2];
            out_UVs.insert(out_UVs.end(), uv, uv + 2);

            uv = &UVs[(uvIndex[2] - 1) * 2];
            out_UVs.insert(out_UVs.end(), uv, uv + 2);

            GLfloat* normal = &normals[(normalIndex[0] - 1) * 3];
            out_normals.insert(out_normals.end(), normal, normal + 3);

            normal = &normals[(normalIndex[1] - 1) * 3];
            out_normals.insert(out_normals.end(), normal, normal + 3);

            normal = &normals[(normalIndex[2] - 1) * 3];
            out_normals.insert(out_normals.end(), normal, normal + 3);


            out_indices.push_back(size);
            out_indices.push_back(size + 1);
            out_indices.push_back(size + 2);*/
        }
    }
}


int main(int argc, const char * argv[]) {
    reader();
    return 0;
}

C++ fast file reading line by line

3 Answers3