Import a growing list() of csv files only to append after imoprting

Question

So I am building a dataset with a growing set of csv's. Rather than adding the new df# = pd.read_csv(filename, index...) I would prefer to just create a function to read the list of csv's and then append them upon importing. Any recommendations? I put the code down below for what I currently have.

import glob
files = glob.glob('*.csv')

files

alg1_2018_2019 = pd.read_csv('alg1_2018_2019.csv', index_col=False)
alg1_2017_2018 = pd.read_csv('alg1_2017_2018.csv', index_col=False)
geometry_2018_2019 = pd.read_csv('geometry_2018_2019.csv', index_col=False)
geom_8_2017_2018 = pd.read_csv('geom_8_2017_2018.csv', index_col=False)
alg2_2016_2017 = pd.read_csv('alg2_2016_2017.csv', index_col=False)
alg1_2016_2017 = pd.read_csv('alg1_2016_2017.csv', index_col=False)
geom_2016_2017 = pd.read_csv('geom_2016_2017.csv', index_col=False)
geom_2015_2016 = pd.read_csv('geom_2015_2016.csv', index_col=False)
alg2_2015_2016 = pd.read_csv('alg2_2015_2016.csv', index_col=False)
alg1_part2_2015_2016 = pd.read_csv('alg1_part2_2015_2016.csv', index_col=False)```

MaxU - stand with Ukraine · Answer 1 · 2019-06-21T10:59:00.137

i'm using the following function:

import pandas as pd
from pathlib import Path

def glob_filemask(filemask):
    """
    allows to "glob" files using file masks with full path

    Usage:
        for file in glob_filemask("/path/to/file_*.txt"):
            # process file here
    or:
        files = list(glob_filemask("/path/to/file_*.txt"))


    :param filemask:    wildcards can be used only in the last part
                        (file name or extension), but NOT in the directory part
    :return:    Pathlib glob generator, for all matching files
    Example:
        glob_filemask("/root/subdir/data_*.csv") -
    will return a Pathlib glob generator for all matching files
        glob_filemask("/root/subdir/single_file.csv") -
    will return a Pathlib glob generator for a single file

    """
    p = Path(filemask)
    try:
        if p.is_file():
            return [p]
    except OSError:
        return p.parent.glob(p.name)

Usage:

df = pd.concat([pd.read_csv(f) for f in glob_filemask("/path/to/file_*.csv")],
               ignore_index=True)

Import a growing list() of csv files only to append after imoprting

1 Answers1