How to read different files from different directories with pattern

Question

I have directories structure:

--- main_dir
------ aaa
--------subaaa
----------file
------ xxx
--------subxxx
----------file
------ 111
--------sub111
----------file

etc.

All directories has different names but FILE has the same name. Every FILE has three lines. My question is how to read these three lines frome every FILE in the fastest way?

I have written code that search for FILEs. But still do not know how to read all of them. Check this out:

import os

def list_files(dir):
    r = []
    for root, dirs, files in os.walk(dir):
        for name in files:
            r.append(os.path.join(root, name))
return r, print(r)
list_files('some_path)

I want to achieve something like that:

import os

def list_files(dir):
    r = []
    for root, dirs, files in os.walk(dir):
        for name in files:
            r.append(os.path.join(root, name))
    return r, print(r)
    with open(r[*]) as f:     #this is the question
    f.readlines()             #how to read all of the files
list_files('some_path)

Possible duplicate of https://stackoverflow.com/questions/2186525/use-a-glob-to-find-files-recursively-in-python — DYZ, Jun 05 '18 at 07:30
not sure, but are you looking for `open` function to read the files once you've the names in a list? check out https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files — Sundeep, Jun 05 '18 at 07:46
This is an output from my code: ['C:\\gpdw-feeds\\1\\DA1\\1.txt', 'C:\\gpdw-feeds\\1\\DA2\\1.txt', 'C:\\gpdw-feeds\\2\\da\\1.txt', 'C:\\gpdw-feeds\\xxx\\da2323232\\1.txt'] So it's just simple list with files in different folders. I want to read all of these '1.txt's lines, because in every 1.txt lines are different. — , Jun 05 '18 at 07:49
yeah, could you try the `open` function, see examples in the docs link I gave and see if that solves your issue — Sundeep, Jun 05 '18 at 07:57

score 1 · Accepted Answer · answered Jun 05 '18 at 12:21

1

To efficiently find every file named 1.txt and print it's contents prefixed with the files name would just be:

find . -name '1.txt' -exec awk '{print FILENAME, $0}' {} +

If that's not all you want then edit your question to clarify.

answered Jun 05 '18 at 12:21

Ed Morton

188,023
17
78
185

score 0 · Answer 2 · answered Jun 05 '18 at 07:36

0

Consider using:

import os

listOfFiles = os.listdir("the absolute path to main_dir")
for fileName in listOfFiles:
 print fileName
 # read the lines of fileName
 with open(fileName) as f:
   contentOfFile = f.readlines()

answered Jun 05 '18 at 07:36

Cezar Cobuz

1,077
1
12
34

It will show only subfolders. Please check my code (I've added to post). Now i have the list of all files but still don't know how to read all of them. – Jun 05 '18 at 07:41

score 0 · Answer 3 · 2018-06-05T08:13:14.210

I found the solution for openning FILEs, here is the code:

import os

def list_files(dir):
    r = []
    for root, dirs, files in os.walk(dir):
        for name in files:
            r.append(os.path.join(root, name))
            with open(os.path.join(root, name)) as f:
                print(f.readlines())
    return r, print(r)


list_files('C:\gpdw-feeds')

Now I have a problem with the structure. How to list ONLY files in the last subfolders? Example:

---main_dir
----sub
-----file1
----sub_sub
-----file2

I want to read only file2, but my code reads all of the files in all of the subs.

How to read different files from different directories with pattern

3 Answers3