Navigate and append csv from child directories

Question

I am in a directory that holds 36 different folders. Each folder has a single csv in it. I want to append each of these together to make a large data frame in python.

in R, I would do this:

cwd = getwd() #get current directory
fil = list.files() #get list of all files/folders in the directory
Bigdf = NULL #initialize empty df
for(i in fil){ #read through all folders in current directory
    setwd(paste0(cwd,'/',i)) #navigate to i'th folder
    fil2 = list.files() #get list of files in i'th folder
    for(j in fil2){
        a = read.csv(paste0(cwd,'/',i,'/',j)) #read in all csv's 
        Bigdf = rbind(Bigdf,a[,c(2,4:11)]) #append desired columns to data frame
    }
    setwd(cwd) 
}

How would I go about doing something like this in python?

I tried to implement How can I read the contents of all the files in a directory with pandas? and How do I list all files of a directory? and to no avail. I think I am missing something obvious, and hope someone can point me in the right direction.

You might want to have a look at the modules `os` (e.g. `os.listdir`) and `csv`. — niko, Apr 05 '19 at 20:47

score 0 · Answer 1 · answered Apr 05 '19 at 20:57

import glob
import pandas as pd

li =[]

for filename in glob.iglob('src/**/*.csv', recursive=True):
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)

Combination of

Import multiple csv files into pandas and concatenate into one DataFrame

and

How to use glob() to find files recursively?

Navigate and append csv from child directories

1 Answers1