C++ vs Python vs Ruby Performance in Listing All Directories Recursively

Question

I just wrote a C++ code to list all the directories in a folder recursively. I'm using Boost Filesystem and I have build the following code statically:

#include <iostream>
#include <boost/filesystem.hpp>
using namespace boost::filesystem;
using namespace std;

int main(int argc, char* argv[]) {
    const path current_file_path(current_path());

    try {
        std::vector<path> directories_vector;
        for (auto&& x : recursive_directory_iterator(current_file_path))
            if (is_directory(x.path()))
                directories_vector.push_back(x.path());
        /* GETTING UP TO HERE TAKES MORE TIME THAN PYTHON OR RUBY*/

        for (auto&& x : directories_vector) {
            cout << x << '\n';
        }
    }
    catch (const filesystem_error& ex) {
        cout << ex.what() << '\n';
    }

    cin.get();
    return 0;
}

I wanted to see how fast this code would work against Python & Ruby. I know I/O related stuff are not good for evaluating code performance but when I run the C++ executable, it takes nearly 3 seconds for 15+ recursive folders while the following Python & Ruby codes are run nearly instantly:

Ruby:

Dir.glob("**/*/")

Python:

[x[0] for x in os.walk(directory)]

All of the codes are running on an SSD. I'm using Visual Studio 2017, Python 3.5.2 and Ruby 2.4 on Windows. The C++ code is using Release/x64 mode and Optimization is set to Maximum Optimization (Favor Speed) (/O2).

Why is the C++ code slower when faced with lots of recursive folders?

Answer 1

By running both the C++ version and the Ruby version with strace we can get some clues why the C++ version is slower.

Using the Linux source code for testing (65000 files):

strace -o '|wc' cpp_recursion
  86417  518501 9463879

strace -o '|wc' ruby -e 'Dir.glob("**/*")' 
  30563  180115 1827588

We see that the C++ version does almost 3x more operations than Ruby.

Looking more closely at the strace output you will find that both programs use getdents to retrieve directory entries, but the C++ version runs lstat on every single file, while the Ruby version does not.

I can only conclude that the C++ version is not implemented as efficiently (or it possibly serves a different purpose) as the Ruby version. The speed difference is not a language issue, but an implementation issue.

N.B. The C++ version with -O optimization runs in 0.347s, while the Ruby version runs in 0.304s. At least on Linux lstat seems to not incur much overhead. Perhaps the situation is different on Windows.