How to open and iterate through a list of CSV files - Python

Question

I have a list of pathnames to CSV files. I need to open each CSV file, take the data without the header and then merge it all together into a new CSV file.

I have this code which gets me the list of CSV file pathnames:

file_list = []
for folder_name, sub_folders, file_names in os.walk(wd):
    for file_name in file_names:
        file_extention = folder_name + '\\' + file_name
        if file_name.endswith('csv'):
            file_list.append(file_extention)

An example of my list is:

['C:\\Users\\Documents\\GPS_data\\West_coast\\Westland\\GPS_data1.csv',
 'C:\\Users\\Documents\\GPS_data\\West_coast\\Westland\\GPS_data2.csv',
 'C:\\Users\\Documents\\GPS_data\\West_coast\\Westland\\GPS_data3.csv']

I am struggling to figure out what to do, any help would be greatly appreciated. Thanks.

Possible duplicate of [Merge CSVs using Python (or Bash)](https://stackoverflow.com/questions/30528673/merge-csvs-using-python-or-bash) — chthonicdaemon, Mar 16 '18 at 03:00
Seams like you need to look up how to open and read a file in Python, this is well documented: https://docs.python.org/3/library/filesys.html — Apeiron, Mar 16 '18 at 03:06
You should not write that nested loop. https://docs.python.org/3/library/glob.html — OneCricketeer, Mar 16 '18 at 03:34

Rusty · Accepted Answer · 2018-03-16T18:03:31.350

2

The main idea is to read in each line of a file, and write it to the new file. But remember to skip the first line that has the column headers in it. I previously recommend the cvs module, however it doesn't seem like that is necessary, since this task does not require analyzing the data.

file_list = ['data1.csv','data2.csv']

with open('new.csv', 'w') as newfile:  # create a new file
    for filename in filelist:
        with open(filename) as csvfile:
            next(csvfile)   # skip the header row
            for row in csvfile:
                newfile.write(line) # write to the new csv file

Edit: clarified my answer.

edited Mar 16 '18 at 18:03

answered Mar 16 '18 at 03:16

Rusty

1,095
9
15

The question seems to be asking about merging files. You're only building a list – OneCricketeer Mar 16 '18 at 03:35
What a legend! This is exactly what I've been trying to do, thanks for the help :) – kels Mar 17 '18 at 02:37

How to open and iterate through a list of CSV files - Python

1 Answers1