How to recursively search for file extension and do a join operation

Question

Our build folder structure looks like something like below:

target file:/home/workspace/slave_1/dock_location/build/project_1/applications/target/*.tar.gz
target file:/home/workspace/slave_1/dock_location/build/project_1/applications/first_app/target/*.tar.gz
target file:/home/workspace/slave_1/dock_location/build/project_2/applications/third_app/target/*.tar.gz

and below snippet gets the list of all the tar.gz file names with the absolute path/

for dir in folders_to_search: 
    # where dir resolves to /home/workspace/slave_1/dock_location/build/project_1/applications/first_app and so on
    final_file = os.path.join(dir, 'target', '*.tar.gz')
    tar_list = glob.glob(final_file) # which will expand and give me the full tar.gz path
    <logic to process the list of tar files...>
    blah blah..

Now, there is a change in folder structure and the updated path looks like

tree /home/workspace/slave_1/dock_location/build/project_1/applications/first_app/target/neo-uber-path/ 

├── design
│   ├── app-lightining
│   │   └── 1.110.23450345
│   │       ├── app-lightining-1.110.23450345.tar.gz
│   │       └── design.json
│   └── background-lightining
│       └── 1.110.23450345
│           ├── background-lightining-1.110.23450345.tar.gz
│           └── design.json
└── features
    ├── provison-robust-feature
    │   └── 1.23.108.35900021
    │       ├── provison-robust-feature-1.1.110.23450345.tar.gz
    │       └── BS.yaml
    └── panda-service
        └── 7.14.610.80350021
            ├── panda-service-1.1.110.23450345.tar.gz
            └── BS.yaml

I'm not sure of the logic to get this line - final_file = os.path.join(dir, 'target', '*.tar.gz') working for the new scenario.

My alternate option is like below, but would like to hear from the experts first, whether a tweak in my existing snippet is possible, rather than writing a new chunk of code..

Kindly correct me if Im wrong and pls. let me know if full piece of code is required , thank you.

Alternate option

for root, dirnames, filenames in os.walk(dir):
    for filename in filenames:
        if filename.endswith(extensions):
            matches.append(os.path.join(root, filename))

I think `os.walk` is the right way to go. Few suggestions: do not use `dir` as a variable since it is a built-in function, and [check for file extension](https://stackoverflow.com/a/541394/13541354) instead of using the less robust `endswith` — ALai, Apr 19 '22 at 10:29
@ALai : thanks ! I'm just exploring whether glob could help me here. `import glob for f in glob.glob('/dir/**/*.tar.ggz', recursive=True): print(f)` — Chel MS, Apr 19 '22 at 10:34

score 0 · Answer 1 · answered Apr 19 '22 at 11:00

0

for file_path in Path(src).rglob('*.tar.gz'):
        print(file_path)

worked.

answered Apr 19 '22 at 11:00

Chel MS

386
1
10

`files = list(Path(src).rglob('*.tar.gz'))`. `Path(src).rglob('*.tar.gz')` is a generator. – Trenton McKinney Apr 19 '22 at 17:15

How to recursively search for file extension and do a join operation

1 Answers1