Iterate over files in multiple directories or an entire hard drive in python

Question

I have script.py file which iterates over specific files in the directory where the script.py file is located.

Script looks something like this:

def my_funct(l):
     Does stuff:
          iterates over list of files
     Does stuff:
globlist = glob.glob('./*.ext')
my_funct(glob list)

I would like to be able to not only iterate over the *.ext files in this directory, but iterate over all the .ext file in all the directories in this directory.

I information I'm reading about os.walk is not making sense.

Thanks.

Post your code including your attempt at adding 'os.walk' and we can provide feedback — thefragileomen, Nov 06 '14 at 22:27
possible duplicate of [Use a Glob() to find files recursively in Python?](http://stackoverflow.com/questions/2186525/use-a-glob-to-find-files-recursively-in-python) — Rafael Barros, Nov 06 '14 at 22:39

espang · Accepted Answer · 2014-11-07T12:39:47.017

3

An example for os.walk. It searches for py files in the folder (and all subfolders) and counts the lines:

# abspath to a folder as a string
folder = '/home/myname/a_folder/'
# in windows:
# folder = r'C:\a_folder\'
# or folder = 'C:/a_folder/'

count = 0
lines = 0
for dirname, dirs, files in os.walk(folder):
    for filename in files:
        filename_without_extension, extension = os.path.splitext(filename)
        if extension == '.py':
            count +=1
            with open(os.path.join(dirname, filename), 'r') as f:
                for l in f:
                    lines += 1
print count, lines

edited Nov 07 '14 at 12:39

answered Nov 06 '14 at 22:56

espang

673
6
6

Thanks for that. I'm unsure what item should be put in the parenthesis next to os.walk. Here you have 'folder'. The documentation is a little weird and other things I've seen are inconsistent with the each other. – mdandr Nov 07 '14 at 12:35
Thanks! That was really slowing me down. – mdandr Nov 07 '14 at 14:44

score 3 · Answer 2 · answered May 12 '16 at 10:31

you can use scandir.walk(path) insted of os.walk(path). it can give faster result compair to os.walk(). this module is included in python35 but you can use python27,python34 using pip install scandir.

my code is here:

import os
import scandir

folder = ' ' #here your dir path
print "All files ending with .py in folder %s:" % folder
file_list = []

for paths, dirs, files in scandir.walk(folder):
#for (paths, dirs, files) in os.walk(folder):
    for file in files:
        if file.endswith(".py"):
            file_list.append(os.path.join(paths, file))

print len(file_list),file_list

scandir.walk() can do exactly what you want in os.walk().

i hope this answer match with your question and here docs for scandir

score 3 · Answer 3 · answered Sep 17 '19 at 20:38

In Python standard library 3.4 and above you can use pathlib

from pathlib import Path

files = Path().cwd().glob("**/*.ext")

It will return you generator with all files that have extension ".ext" in current and child directories. You can iterate then over those files

for f in files:
    print(f)
    # do other stuff

or you can do in one line:

for f in Path().cwd().glob("../*.ext"):
    print(f)
    # do other stuff

Iterate over files in multiple directories or an entire hard drive in python

3 Answers3