how to read mutliple csv files and store them in different dataframe?

Question

Say I have 200 csv files, I want to read these csv files at one time, and store each csv file in different data frames like df1 for the first file and so on up to df200. Doing manual like df1=pd.read_csv takes a lot of time up to 200. How do I do this using pandas?

I have tried using for loop, but unable to approach, stuck.

This answer shows you [how to read many files in a dictionary](https://stackoverflow.com/a/56990861/2641825). — Paul Rougieux, Dec 13 '19 at 07:13

Dishin H Goyani · Accepted Answer · 2020-04-22T12:32:51.330

3

import pandas as pd
import glob

all_files = glob.glob("file_path" + "/*.csv")

dfs_dict = {}

for idx, filename in enumerate(all_files):
    df = pd.read_csv(filename, index_col=None, header=0)
    dfs_dict["df" + str(idx)] = df

edited Apr 22 '20 at 12:32

answered Dec 13 '19 at 07:18

Dishin H Goyani

7,195
3
26
37

score 2 · Answer 2 · answered Dec 13 '19 at 07:14

Try using this :

import pandas as pd
import glob

path = r'path of the folder where all csv exists' 
all_files = glob.glob(path + "/*.csv")

li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

li will have all the csv's... you can furthur preprocess them to separate them into different files,

or if all the csv's have the same column and you want to concatenate them to a single dataframe, you could use the concat function in pandas over li to return the single dataframe.

score 1 · Answer 3 · answered Dec 13 '19 at 07:14

1

import pandas as pd
import os

dfs=[] #empty list of dataframes

dirname = #where your files are

for root,folders,files in os.walk(dirname):
    for file in files:
        fp = os.path.join(root,file)
        df=pd.read_csv(fp)
        dfs.append(df)

df=pd.concat(dfs)

answered Dec 13 '19 at 07:14

Ajay Shah

414
5
10

1

It's not the best idea to create a temporary list of DFs. You can pass a generator expression to concat, or make a standalone generator, if it's too complex for a single expression. Something along these lines: `df = pd.concat((pd.read_csv(f) for f in files))` – Oleg O Dec 13 '19 at 09:26

how to read mutliple csv files and store them in different dataframe?

3 Answers3