How can i loop through each of the csv file in a folder,pick certain columns of each file and merge it with an already existing .csv file

Question

I have a folder that has multiple .csv files. I'm trying to loop through each of the csv file,pick certain columns and merge it with an already existing file. can anyone help me in this regard......

Answer 1

try this:

If you have same columns in all your csv files then you can try the code below. I have added header=0 so that after reading csv first row can be assigned as the column names.

import pandas as pd
import glob

path = r'C:\TEST' # use your path
all_files = glob.glob(path + "/*.csv")

li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)

Check this answer for more information: Import multiple csv files into pandas and concatenate into one DataFrame