python, pandas, csv import and more

Question

I have seen many questions in regards to importing multiple csv files into a pandas dataframe. My question is how can you import multiple csv files but ignore the last csv file in your directory? I have had a hard time finding the answer to this.

Also, lets assume that the csv file names are all different which is why the code file is "/*.csv"

any resource would also be greatly appreciated. Thank you!

path =r'C:\DRO\DCL_rawdata_files' # use your path
allFiles = glob.glob(path + "/*.csv")
frame = pd.DataFrame()
list_ = []
for file_ in allFiles:
    df = pd.read_csv(file_,index_col=None, header=0)
    list_.append(df)
frame = pd.concat(list_)

This is also assuming you know exactly what the *last* file is. — Leb, Mar 16 '16 at 16:33
if you would answer yourself the following question: _last when sorted by `what` (creation time, file name, file size, etc.)?_, then you would find an answer pretty easily, i guess... — MaxU - stand with Ukraine, Mar 16 '16 at 16:39
@MaxU very good point, I did not realize that. Let's base it on creation time if possible (as in the most recent csv file) — antonio_zeus, Mar 16 '16 at 17:06

score 1 · Accepted Answer · answered Mar 16 '16 at 18:30

1

Try this:

import os
import glob
import pandas as pd

def get_merged_csv(flist, **kwargs):
    return pd.concat([pd.read_csv(f, **kwargs) for f in flist], ignore_index=True)

path =r'C:\DRO\DCL_rawdata_files' # use your path
fmask = os.path.join(path, '*.csv')
allFiles = sorted(glob.glob(fmask), key=os.path.getmtime)

frame = get_merged_csv(allFiles[:-1], index_col=None, header=0)

answered Mar 16 '16 at 18:30

MaxU - stand with Ukraine

205,989
36
386
419

thank you for the response. It is great. Can I ask you another question? Let's assume that in addition to skipping the most recent CSV file that I wanted to loop through the list n-periods. As in, import the CSV files 4 at a time AND assume that I don't know how many CSV there are (could be even or odd) - ty – antonio_zeus Mar 21 '16 at 21:58
@antonio_zeus, do you want to load your CSV's in chunks, having 4 files in one chunk? – MaxU - stand with Ukraine Mar 22 '16 at 09:31
let's use the code from above. Let's say allFiles has over 80 csv files. How do I loop 4 at a time into for file_ in allFiles regardless of how many there are? Ex: 82 csv files, would loop in 20 times at 4 files each and then loop in the remaining 2 at the end as well. Is this possible? ty! – antonio_zeus Mar 22 '16 at 13:22
@antonio_zeus, you want to read [this](http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python) – MaxU - stand with Ukraine Mar 22 '16 at 13:26
thank you for pointing me in the right direction - I really appreciate it. – antonio_zeus Mar 22 '16 at 13:30
@antonio_zeus, glad i could help :) – MaxU - stand with Ukraine Mar 22 '16 at 13:32

python, pandas, csv import and more

1 Answers1