Loop through all .csv files in folder

Question

I'm having trouble expanding my code to use on a folder of .csv's as opposed to just one file. I have written the code to read and manipulate a single file which starts with the following code and is putting data into 3 lists:

with open('Filename_raw.csv', 'r') as read_obj:
    csv_reader = reader(read_obj)
    list1 = []
    list2 = []
    list3 = []
    for row in csv_reader:
        if len(row) == 3:
        list1.append(row)
        if len(row) == 16:
        list2.append(row)
        if len(row) == 21:
        list3.append(row)
    ...
    ...
    ...

I had to run through each csv line by line because of its structure(3 different tables in one csv)

I have a file path C:\Users\xxxx\xxxxxx\xxxxxx\xxxxxxxx where the .csvs are located and so im hoping for some guidance on how to loop through all files in folder as opposed to just one.

Filenames are similarly structured and always end in "_raw.csv"

Any help would be much appreciated - Thanks!

Does this answer your question? [Read in all csv files from a directory using Python](https://stackoverflow.com/questions/33503993/read-in-all-csv-files-from-a-directory-using-python) — shivankgtm, Apr 26 '20 at 10:49
Check out `os.walk()` https://docs.python.org/3/library/os.html#os.walk or the simpler `glob.iglob()` https://docs.python.org/3/library/glob.html#glob.iglob — Todd, Apr 26 '20 at 10:55

Nandu Raj · Accepted Answer · 2020-04-26T11:15:25.540

The following code should work. I have also updated code as instead of using three if conditions, you can use if - elif. Else the code will check the three conditions, even if the first condition satisfies which is unnecessary.

import os

directory = os.path.join("c:\\","path")
for root,dirs,files in os.walk(directory):
    for file in files:
       if file.endswith("_raw.csv"):
          with open(file, 'r') as read_obj:
          csv_reader = reader(read_obj)
          list1 = []
          list2 = []
          list3 = []
          for row in csv_reader:
             if len(row) == 3:
                list1.append(row)
             elif len(row) == 16:
                list2.append(row)
             elif len(row) == 21:
                list3.append(row)

Thanks heaps!. I ended up using the `glob` function which worked and so did this! — SOK, Apr 26 '20 at 11:02

Loop through all .csv files in folder

1 Answers1