Pyhton csv reader: loop over csv files

Question

I have several csv files that are named: list_csv_X with X going from 0 to 5. I would like to do a loop to read them with Python.

I am currently doing it manually:

for filepath in list_csv_0:
    with open(filepath,'r',encoding="utf8", errors='ignore') as csvfile:
        BLABLABLA
for filepath in list_csv_1:
    with open(filepath,'r',encoding="utf8", errors='ignore') as csvfile:
        BLABLABLA
for filepath in list_csv_2:
    with open(filepath,'r',encoding="utf8", errors='ignore') as csvfile:
        BLABLABLA

How can I do it?

Thanks a lot!

You could use an f-string to construct the filenames in a loop — DarkKnight, Mar 21 '23 at 12:18
use pyhton glob or Path.glob with a file pattern "lst_csv_*.csv" in order to get all csv files. then open them as you do... — Je Je, Mar 21 '23 at 12:32

DarkKnight · Answer 1 · 2023-03-21T15:48:51.567

4

Use a simple for loop and an f-string as follows:

def process(filename):
    with open(filename) as indata:
        pass # process file here

for fileno in range(6):
    with open(f'list_csv_{fileno}') as catalogue:
        for line in catalogue:
            process(line.strip())

edited Mar 21 '23 at 15:48

answered Mar 21 '23 at 12:21

DarkKnight

19,739
3
6
22

This solution does not work in my case. I think that I need an extra loop: in my case, each `list_csv_X` file contain a list of path toward csv file that have different name: For instance `list_csv_0` contains `/home/path0/x01_200000.csv /home/path0/x02_200000.csv /home/path0/x03_200000.csv` and `list_csv_1` contains `/home/path1/x01_300000.csv /home/path1/x02_300000.csv /home/path1/x03_300000.csv` Both the path and the file name are modified. – Martin7 Mar 21 '23 at 14:06
1

@Martin7 That wasn't clear (to me) in your original question. See edit. This kind of pattern is often well-suited to multiprocessing/multithreading otherwise each file will be processed linearly. Something for you to think about – DarkKnight Mar 21 '23 at 15:30

ProfDFrancis · Accepted Answer · 2023-03-21T12:27:03.680

If I understand correctly, each of your 6 items is a list of filenames, yes?

I am assuming that, because you are doing this, and saying that it currently works:

for filepath in list_csv_0:

That would lead to an error, if list_csv_0 were a filename, because filepath values would be individual characters of the filename, not ever a whole filename.

You can loop over the lists


list_csvs = [list_csv_0,list_csv_1,list_csv_2,list_csv_3,list_csv_4,list_csv_5]

for list_csv in list_csvs:
    for filepath in list_csv:
        with open(filepath,'r',encoding="utf8", errors='ignore') as csvfile:
            BLABLABLA

Pyhton csv reader: loop over csv files

2 Answers2

If I understand correctly, each of your 6 items is a list of filenames, yes?

You can loop over the lists