Iterating through directories with Python

Question

I need to iterate through the subdirectories of a given directory and search for files. If I get a file I have to open it and change the content and replace it with my own lines.

I tried this:

import os

rootdir ='C:/Users/sid/Desktop/test'

for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        f=open(file,'r')
        lines=f.readlines()
        f.close()
        f=open(file,'w')
        for line in lines:
            newline = "No you are not"
            f.write(newline)
        f.close()

but I am getting an error. What am I doing wrong?

Answer 1

The actual walk through the directories works as you have coded it. If you replace the contents of the inner loop with a simple print statement you can see that each file is found:

import os
rootdir = 'C:/Users/sid/Desktop/test'

for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        print(os.path.join(subdir, file))

Answer 2

Another way of returning all files in subdirectories is to use the pathlib module, introduced in Python 3.4, which provides an object oriented approach to handling filesystem paths (Pathlib is also available on Python 2.7 via the pathlib2 module on PyPi):

from pathlib import Path

rootdir = Path('C:/Users/sid/Desktop/test')
# Return a list of regular files only, not directories
file_list = [f for f in rootdir.glob('**/*') if f.is_file()]

# For absolute paths instead of relative the current dir
file_list = [f for f in rootdir.resolve().glob('**/*') if f.is_file()]

Since Python 3.5, the glob module also supports recursive file finding:

import os
from glob import iglob

rootdir_glob = 'C:/Users/sid/Desktop/test/**/*' # Note the added asterisks
# This will return absolute paths
file_list = [f for f in iglob(rootdir_glob, recursive=True) if os.path.isfile(f)]

The file_list from either of the above approaches can be iterated over without the need for a nested loop:

for f in file_list:
    print(f) # Replace with desired operations

Answer 3

From python >= 3.5 onward, you can use **, glob.iglob(path/**, recursive=True) and it seems the most pythonic solution, i.e.:

import glob, os

for filename in glob.iglob('/pardadox-music/**', recursive=True):
    if os.path.isfile(filename): # filter dirs
        print(filename)

Output:

/pardadox-music/modules/her1.mod
/pardadox-music/modules/her2.mod
...

---

Notes:

1. glob.iglob

   glob.iglob(pathname, recursive=False)

   Return an iterator which yields the same values as glob() without actually storing them all simultaneously.

2. If recursive is True, the pattern '**' will match any files and zero or more directories and subdirectories.

3. If the directory contains files starting with . they won’t be matched by default. For example, consider a directory containing card.gif and .card.gif:

   >>> import glob
   >>> glob.glob('*.gif') ['card.gif'] 
   >>> glob.glob('.c*')['.card.gif']
   

4. You can also use rglob(pattern), which is the same as calling glob() with **/ added in front of the given relative pattern.