How can we read all json files from all sub directory?

Question

I want to read lots of json files from multiple subdirectory at one step. How can I do that?

import glob
hit = glob.glob("/./*.json")

Does this answer your question? [Python read JSON files from all sub-directories](https://stackoverflow.com/questions/56866697/python-read-json-files-from-all-sub-directories) — Tanmaya Meher, Jul 10 '21 at 12:38
Does this answer your question? [Recursive sub folder search and return files in a list python](https://stackoverflow.com/questions/18394147/recursive-sub-folder-search-and-return-files-in-a-list-python) — mujjiga, Jul 10 '21 at 12:51

score 4 · Accepted Answer · answered Jul 10 '21 at 13:19

You can get all file paths by a small modification in your original code.

From Python docs

If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories.

from glob import glob
hits = glob("**/*.json", recursive=True)

"""
You will get a list of paths like this
hits = [
#....
'<path>/delta/job-3/33ce4079-c8db-11eb-8683-f30d80ef99b2.json',
 '<path>/delta/pair-1/cf81188b-c8d7-11eb-8367-5fb2efe63fc6.json',
 '<path>/sub/b1d91522-cab4-11eb-a5cb-113c64720fe0.json',
 '<path>/sub/979a7c2b-cab4-11eb-a5cb-b91d90206530.json',
 '<path>/sub/977e4199-cab4-11eb-a5cb-33b60824fb94.json',
 '<path>/sub/a5fb35cd-cab4-11eb-a5cb-5f6cd57276ff.json',
 '<path>/sub/a60520de-cab4-11eb-a5cb-2723d138a03b.json',
 '<path>/sub/9805e82c-cab4-11eb-a5cb-3987a3853fca.json'
]
"""

score 0 · Answer 2 · answered Jul 10 '21 at 13:03

you can use the logic below to retrieve a list of filepaths.

This answer has a stupid amount of ways to do this.

Once you have the filelist loading them is very simple...

import os
import json


def get_files_from_path(path: str='.', extension: str=None) -> list:
    """return list of files from path"""
    # see the answer on the link below for a ridiculously 
    # complete answer for this. I tend to use this one.
    # note that it also goes into subdirs of the path
    # https://stackoverflow.com/a/41447012/9267296
    result = []
    for subdir, dirs, files in os.walk(path):
        for filename in files:
            filepath = subdir + os.sep + filename
            if extension == None:
                result.append(filepath)
            elif filename.lower().endswith(extension.lower()):
                result.append(filepath)
    return result


filelist = get_files_from_path(extension='.json')
jsonlist = []
for filepath in filelist:
    with open(filepath) as infile:
        jsonlist.append(json.load(infile))

from pprint import pprint
pprint(jsonlist)

How can we read all json files from all sub directory?

2 Answers2