Using python os.walk how to check for directory name and only process those files (recursively) in the specific directory?

Question

I'm using os.walk to run through directory "foo". I want to process .dat files but how to check for a directory name and only process the specific directory?

If dir="bar" then process files.dat. Do not process "notbar". I'm probably missing something simple

 C:\data\foo
       - notbar
           -123
             -file1.dat
           -456
             -file2.dat
             -file3.dat
       - bar
           -123
             -file1.dat
           -456
             -file2.dat
             -file3.dat

this finds all .dat files....

    for (root, dirnames, filenames) in os.walk(base_path):
        print('Found directory: {0}'.format(root))
        for filename in filenames:
            if filename.endswith(".dat"):
                print(filename)

To add to @juanpa.arrivillaga, use `os.walk(os.path.join(base_path, "bar"))` for compatibility with multiple OS (windows uses \ and unix /, not our problem to deal with that) — Seraf, Sep 30 '20 at 14:46
That is not really the option, this is a simplified example. The user will specify a root directory and not the path to the known folder name. — rdebruyn, Sep 30 '20 at 14:48
@Seraf I would personally recommend the nicer `pathlib` instead of `os.path`, so then it's just `Path(base_path) / 'bar'` — juanpa.arrivillaga, Sep 30 '20 at 14:48
@rdebruyn so, there is some directory *somewhere* called `bar`, that is all you know? — juanpa.arrivillaga, Sep 30 '20 at 14:48
@juanpa.arrivillaga yes. There will be always be a directory in the structure with a known name ie "bar" — rdebruyn, Sep 30 '20 at 14:51
Well, then you can first find `bar`, then just walk again from `bar` — juanpa.arrivillaga, Sep 30 '20 at 14:53

score 1 · Answer 1 · answered Sep 30 '20 at 22:52

glob is really good for this. It returns all the files that match a certain pattern.

There is a reference for the patterns, but the most useful are:

* matches everything except path slashes (\ for windows, / for mac / linux)
** matches zero or more directories

In your example, you want to find the .dat (*.dat) files in any sub-directory (*) of a sub-directory (bar) inside a base path base_path. To get these files we can write

from glob import glob

filenames = glob(base_path + "\\bar\\*\\*.dat")

It is better to use os.path.join for cross-platform:

from glob import glob

filenames = glob(os.path.join(base_path, "bar", "*", "*.dat"))

Check out the results here

If bar is not necessarily the immediate sub-directory of base_path, but nested further down, you could use **:

from glob import glob

filenames = glob(os.path.join(base_path, "**", "bar", "*", "*.dat"))

Finally, glob will not necessarily return the files in any order. To get them in alphabetical order use sorted(filenames). To get them in modified order use sorted(filenames, key=os.path.getmtime) as per this answer.

score 0 · Answer 2 · answered Jun 14 '23 at 10:52

As stated in the comment, a possible solution can be to do a second os.walk. In detail

databasePath = ".\database"

import os

for (root, dirs, files) in os.walk(databasePath):
    for dir in dirs:
    if dir == "myLabel":
        for (_root, _dirs, _files) in os.walk(os.path.join(root, dir)):
            i = 0
            print(_files)

Using python os.walk how to check for directory name and only process those files (recursively) in the specific directory?

2 Answers2

Linked