Loop through each subdirectory in a main directory and run code against each file using OS

Question

Essentially what I'm trying to do is loop through a directory that contains multiple sub-directories and within those run code against each file in a for loop.

The only start I managed to make was listing the directories but as I've rarely ever used os I'm not sure if I could potentially loop through os.chdir and a bit of f string formatting to loop through each subdirectory.

The files I want to run code against are just txt files.

Here goes my code, up to the moment:

import os 

for folders in os.listdir('../main_directory'):
    for something in os.listdir(f'{folders}'):
        # run some function of sorts
        pass

Any help would be greatly appreciated.

[os-walk-to-recursively-traverse-directories-in-python](https://stackoverflow.com/questions/16953842/using-os-walk-to-recursively-traverse-directories-in-python) Please check this it might solve your problem — Manjari, Jul 14 '22 at 14:09

Rodrigo Laguna · Accepted Answer · 2022-07-14T14:52:34.360

I like using pure os:

import os

for fname in os.listdir(src):

    # build the path to the folder
    folder_path = os.path.join(src, fname)
    
    if os.path.isdir(folder_path):
        # we are sure this is a folder; now lets iterate it
        for file_name in os.listdir(folder_path):
            file_path = os.path.join(folder_path, file_name)
            # now you can apply any function assuming it is a file
            # or double check it if needed as `os.path.isfile(file_path)`

Note that this function just iterate over the folder given at src and one more level:

src/foo.txt  # this file is ignored
src/foo/a.txt  # this file is processed
src/foo/foo_2/b.txt  # this file is ignored; too deep.
src/foo/foo_2/foo_3/c.txt  # this file is ignored; too deep.

In case you need to go as deep as possible, you can write a recursive function and apply it to every single file, as follows:

import os

def function_over_files(path):
    if os.path.isfile(path):
        # do whatever you need with file at path
    else:
        # this is a dir: we will list all files on it and call recursively
        for fname in os.listdir(path):
            f_path = os.path.join(path, fname)

            # here is the trick: recursive call to the same function
            function_over_files(f_path)

src = "path/to/your/dir"
function_over_files(src)

This way you can apply the function to any file under path, don't care how deep it is in the folder, now:

src/foo.txt  # this file is processed; as each file under src
src/foo/a.txt  # this file is processed
src/foo/foo_2/b.txt  # this file is processed
src/foo/foo_2/foo_3/c.txt  # this file is processed

That would work nicely. Suggestion to OP, to also incorporate additionally another logic layer after the if statement to parse files in the base layer of folder (ie. in the form of else/elif statement). This will make your code more robust. I differ to your specific use-case and folder structure, of course. — TangibleTech, Jul 14 '22 at 14:53

Denis Gonçalves · Answer 2 · 2022-07-14T14:17:48.213

0

You could try something like this:

for subdir, dirs, files in os.walk(rootdir):

Now you have "access" to all subdirs, dirs, and files for your main folder.

Hope it helps

edited Jul 14 '22 at 14:17

answered Jul 14 '22 at 14:17

Denis Gonçalves

111
4

Loop through each subdirectory in a main directory and run code against each file using OS

2 Answers2