Python: Loop to open multiple folders and files in python

Question

I am new to python and currently work on data analysis.

I am trying to open multiple folders in a loop and read all files in folders. Ex. working directory contains 10 folders needed to open and each folder contains 10 files.

My code for open each folder with .txt file;

file_open = glob.glob("home/....../folder1/*.txt")

I want to open folder 1 and read all files, then go to folder 2 and read all files... until folder 10 and read all files. Can anyone help me how to write loop to open folder, included library needed to be used?

I have my background in R, for example, in R I could write loop to open folders and files use code below.

folder_open <- dir("......./main/")
for (n in 1 to length of (folder_open)){
    file_open <-dir(paste0("......./main/",folder_open[n]))

    for (k in 1 to length of (file_open){
        file_open<-readLines(paste0("...../main/",folder_open[n],"/",file_open[k]))
        //Finally I can read all folders and files.
    }
}

Does [this](https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory) help? — GalAbra, Mar 06 '18 at 16:48
None of the answers actually answer the question! The question is about a specific list of directories from folder1 to folder10, not all directories (of which there could be thousands). — PhilHibbs, Jun 06 '22 at 10:19

GalAbra · Accepted Answer · 2020-11-06T05:30:46.707

5

This recursive method will scan all directories within a given directory and then print the names of the txt files. I kindly invite you to take it forward.

import os

def scan_folder(parent):
    # iterate over all the files in directory 'parent'
    for file_name in os.listdir(parent):
        if file_name.endswith(".txt"):
            # if it's a txt file, print its name (or do whatever you want)
            print(file_name)
        else:
            current_path = "".join((parent, "/", file_name))
            if os.path.isdir(current_path):
                # if we're checking a sub-directory, recursively call this method
                scan_folder(current_path)

scan_folder("/example/path")  # Insert parent direcotry's path

edited Nov 06 '20 at 05:30

answered Mar 06 '18 at 17:12

GalAbra

5,048
4
23
42

1

os.walk does the same. – Reck Mar 06 '18 at 17:15
I agree, it is similar. Personally I believe it's better to practice and use imperative code before embracing built-in functions. – GalAbra Mar 06 '18 at 17:18
1

Yeah for practice its better. For applications we may choose not to reinvent the wheel. – Reck Mar 06 '18 at 18:36
Point taken. Have a nice evening – GalAbra Mar 06 '18 at 18:47

score 2 · Answer 2 · answered Mar 06 '18 at 17:19

Given the following folder/file tree:

C:.
├───folder1
│       file1.txt
│       file2.txt
│       file3.csv
│
└───folder2
        file4.txt
        file5.txt
        file6.csv

The following code will recursively locate all .txt files in the tree:

import os
import fnmatch

for path,dirs,files in os.walk('.'):
    for file in files:
        if fnmatch.fnmatch(file,'*.txt'):
            fullname = os.path.join(path,file)
            print(fullname)

Output:

.\folder1\file1.txt
.\folder1\file2.txt
.\folder2\file4.txt
.\folder2\file5.txt

score 1 · Answer 3 · answered Mar 06 '18 at 17:27

Your glob() pattern is almost correct. Try one of these:

file_open = glob.glob("home/....../*/*.txt")
file_open = glob.glob("home/....../folder*/*.txt")

The first one will examine all of the text files in any first-level subdirectory of home/......, whatever that is. The second will limit itself to subdirectories named like "folder1", "folder2", etc.

I don't speak R, but this might translate your code:

for filename in glob.glob("......../main/*/*.txt"):
    with open(filename) as file_handle:
        for line in file_handle:
            # perform data on each line of text

score 0 · Answer 4 · edited Mar 06 '18 at 17:11

0

I think nice way to do that would be to use os.walk. That will generate tree and you can then iterate through that tree.

import os
directory = './'
for d in os.walk(directory):
    print(d)

edited Mar 06 '18 at 17:11

Reck

1,388
11
20

answered Mar 06 '18 at 16:49

Alex

731
1
6
21

Why a comment is given as an answer. There is no answer. Thats the suggestion you are giving. – Reck Mar 06 '18 at 16:56
@Reck please suggest a way to turn this comment into an answer. – Alex Mar 06 '18 at 16:59
Add some code to make your suggestion as an answer. – Reck Mar 06 '18 at 17:02
There is no button as such to turn this comment into an answer or else I would have surely helped you. – Reck Mar 06 '18 at 17:03
The question is not about listing ALL directories, just the directories "folder1" to "folder10". – PhilHibbs Jun 06 '22 at 10:20

score 0 · Answer 5 · answered Mar 06 '18 at 17:35

This code will look for all directories inside a directory, printing out the names of all files found there:

#--------*---------*---------*---------*---------*---------*---------*---------*
# Desc: print filenames one level down from starting folder
#--------*---------*---------*---------*---------*---------*---------*---------*

import os, fnmatch, sys

def find_dirs(directory, pattern):
    for item in os.listdir(directory):
        if os.path.isdir(os.path.join(directory, item)):
            if fnmatch.fnmatch(item, pattern):
                filename = os.path.join(directory, item)
                yield filename


def find_files(directory, pattern):
    for item in os.listdir(directory):
        if os.path.isfile(os.path.join(directory, item)):
            if fnmatch.fnmatch(item, pattern):
                filename = os.path.join(directory, item)
                yield filename



#--------*---------*---------*---------*---------*---------*---------*---------#
while True:#                       M A I N L I N E                             #
#--------*---------*---------*---------*---------*---------*---------*---------#
#                                  # Set directory
    os.chdir("C:\\Users\\Mike\\\Desktop")

    for filedir in find_dirs('.', '*'):
        print ('Got directory:', filedir)
        for filename in find_files(filedir, '*'):
            print (filename)

    sys.exit() # END PROGRAM

score 0 · Answer 6 · answered Nov 06 '20 at 05:50

0

pathlib is a good choose

from pathlib import Path

# or use: glob('**/*.txt')
for txt_path in [_ for _ in Path('demo/test_dir').rglob('*.txt') if _.is_file()]:
    print(txt_path.absolute())

answered Nov 06 '20 at 05:50

Carson

6,105
2
37
45

Python: Loop to open multiple folders and files in python

6 Answers6