read all files in sub folder with pandas

Question

My notebook is in the home folder where I also have another folder "test". In the test folder, I have 5 sub folders. Each of the folder contains a .shp file. I want to iterate in all sub folders within test and open all .shp files. It doesn't matter if they get overwritten.

data = gpd.read_file("./test/folder1/file1.shp")
data.head()

How can I do so? I tried this

path = os.getcwd()
files = glob.glob(os.path.join(path + "/test/", "*.shp"))
print(files)

but this would only go in 1 layer deep.

what about just passing `path + "/test/"` to `path.join`? e: Tested and works for me — Ben, Aug 05 '21 at 15:15

score 3 · Accepted Answer · answered Aug 05 '21 at 15:20

3

you can use the os.walk method in the os library.

import os
import pandas as pd
for root, dirs, files in os.walk("./test"):
    for name in files:
        fpath = os.path.join(root, name)
        data = pd.read_file(fpath)

answered Aug 05 '21 at 15:20

Jett Chen

382
2
8

score 0 · Answer 2 · answered Aug 05 '21 at 15:26

0

Just do os.chdir(path), and then use glob.glob(os.path.join('*.shp')). It should work. You have already given the string to join 'os.path'.

answered Aug 05 '21 at 15:26

Rockoos

37
8

Is this for the updated qs? How do I iterate within the folders inside test? – x89 Aug 05 '21 at 15:38
For your current qs, u could do what I said. It should work. – Rockoos Aug 05 '21 at 15:49

read all files in sub folder with pandas

2 Answers2

Linked