How to read huge file into vector or array

Question

I made code that read file as vector. But it's terribly slow.(it takes 12sec to read about 430000lines) What's wrong with my code? When I do same thing in C# it takes only 0.5 ~ 1 sec.

#include <iostream>
#include <string>
#include <fstream>
#include <vector>

using namespace std;

bool getFileContent(string fileName, vector<string> &vecOfStrs)
{
    ifstream in(fileName.c_str());

    if (!in)
    {
        std::cerr << "Cannot open the File : " << fileName << std::endl;

        return false;
    }

    string str;

    while (getline(in, str))
    {
        if (str.size() > 0)
        {
            vecOfStrs.push_back(str);
        }
    }

    in.close();

    return true;
}

int main()
{
    string my_file_path = "C:/Users/user/Desktop/myfile.txt";
    vector<string> lines;
    bool result = getFileContent(my_file_path, lines);

    if (result)
    {
        cout << lines.capacity() << endl;
    }
}

How are you compiling/running. Vectors can be slow if there is iterator debugging — MDK, Feb 02 '19 at 02:35
Have you tried compiling it with like level 3 (-O3) optimizations? Also, [`capacity()`](https://en.cppreference.com/w/cpp/container/vector/capacity) will give you number of strings that the container has allocated memory for. To get the actual number of strings use [`size()`](https://en.cppreference.com/w/cpp/container/vector/size). — atru, Feb 02 '19 at 02:37
Switch to the release build for timing. I have seen cases where the Debug build was 100 times slower than Release with Visual Studio. — drescherjm, Feb 02 '19 at 02:48
If you have a (rough) idea of how many lines to expect, you can `reserve()` the vector's capacity before then iterating through the file. — Remy Lebeau, Feb 02 '19 at 02:49
@Busted I don't really use/know Visual Studio except an outdated (2010) version for one side project. I managed to find [the optimizations](https://learn.microsoft.com/en-us/cpp/build/reference/o1-o2-minimize-size-maximize-speed?view=vs-2017) with [this documentation](https://learn.microsoft.com/en-us/cpp/build/reference/setting-compiler-options?view=vs-2017). I see that by default they're disabled (at least in my case) and that there is a "full optimization" option that you may want to try. — atru, Feb 02 '19 at 02:56
It probably isn’t the largest part of the cost, but you can use `std::move` with `push_back` to avoid copying each line only to replace the original with the next. — Davis Herring, Feb 02 '19 at 03:01

Loc Tran · Accepted Answer · 2019-02-02T05:14:09.120

I assumed you are using Visual Studio for developing your application. Do the following steps for Optimzation is /O2

1> Project Property --> Configuration Properties --> C/C++ --> Code Generation --> Basic Runtime Checks = Default

2> Project Property --> Configuration Properties --> C/C++ --> Optimization --> Optimization = Maximum Optimization (Favor Speed) (/O2)

It will make your program is optimized at run time at maximum level. If it still not good, I think you should count the number of line in file with this link How to count lines of a file in C++?

and reserve this number for initial vector's capacity.

Hope it will resolve your issue. Here is my solution

ifstream in("result_v2.txt");
vector<string> lines;
string str;

while (std::getline(in, str))
{
    if (str.size() > 0)
    {
        lines.push_back(str);
    }
}

in.close();

I am not sure you want to modify the Debug build to enable optimization when you should have a Release build that you can easily switch to. — drescherjm, Feb 02 '19 at 13:27

How to read huge file into vector or array

1 Answers1