Optimizing a function that returns paths of a folder in python

Question

I am looking to write a function that looks into a given directory and gives me a list of paths of all folders of a particular name.

Let's say I need to search the Desktop and all its subdirectories for folders named "Test". The original code to do it is this:

def finder():
    lst = []
    for root, dir, files in os.walk(r'C:\Users\username\Desktop'):
        for i in dir:
            if i == 'Test':
                lst.append(os.path.join(root,i))
    return lst

I looked online and found that list comprehensions can be much faster in such cases and came up with this function:

def finder2():
    lst = [i[0] for i in os.walk(r'C:\Users\username\Desktop') if i[0][-4:]=='Test']
    return lst

I timed both functions using timeit for 100 repetitions and found that they take a similar amount of time.

Why is the list comprehension not faster?
How can I make it faster?
Is there any other faster way to do the same thing?

Thanks!

in my experience list comprehension is not faster, it just looks nicer and concise — Reza, Sep 04 '20 at 20:36
I heard that it can be upto 50% faster because there is no need to "append" everytime — Nikit Parakh, Sep 04 '20 at 20:37
Does this answer your question? [How do I list all files of a directory?](https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory) — Trenton McKinney, Sep 04 '20 at 20:37
This post does not really acknowledge speed of the code unfortunately — Nikit Parakh, Sep 04 '20 at 20:41

score 1 · Accepted Answer · answered Sep 04 '20 at 20:49

The task is probably mainly I/O limited so you are unlikely to achieve much speed-up however it is performed.

List comprehensions are still effectively loops in the Python level, and may be slightly faster than a for loop because the append attribute does not need to be looked up each time, but the difference is not normally very significant.

To take a more radical comparison, on a Linux system I compared the timings of your Python code with the equivalent find command (find /starting/directory -type d -name Test). Here, find is an executable compiled from C code, so for CPU limited tasks would be expected to be significantly quicker than any explicit loops in Python (including list comprehensions). In fact, I found that running find was only on average about 25% quicker than the Python code. This is indicative of the fact the task is I/O limited, and you are unlikely to achieve significant speed-up by changing the algorithm.

Thank you for this information, I had been scratching my head for alternatives for quite a while! — Nikit Parakh, Sep 04 '20 at 20:50

Optimizing a function that returns paths of a folder in python

1 Answers1