Python - importing and re-writing multiple CSV files

Question

I have a number of csv files that I would like to read, remove the empty columns and rewrite each to a separate file.

So far in Jupyter Notebook I can do this for a single file without any problems:

from pandas.io.parsers import read_csv
data = read_csv('C:\\Users\\tom\\Desktop\\plugs\\20_80-1_plugs.csv')
filtered_data = data.dropna(axis='columns', how='all')
filtered_data.to_csv('C:\\Users\\tom\\Desktop\\plugs\\20_80-1_plugs_modify.csv'), index=False)

Can anyone help with extending this to read in the next csv file in the folder, and write with the same name as the input (say with _modify in the file name). The folder will comprise of files such as:

20_80-1_plugs.csv
11_91-9_plugs.csv
55_21-2_plugs.csv
etc

check out: https://stackoverflow.com/questions/33503993/read-in-all-csv-files-from-a-directory-using-python for tips or iterating through the list of csv files in a folder. — JonSG, Feb 22 '23 at 16:00

score 2 · Accepted Answer · edited Feb 22 '23 at 20:02

If you want to edit all the files in the folder you can do

import os
import pandas as pd

root_folder = 'C:\\Users\\tom\\Desktop\\plugs\\'

for file in os.listdir(root_folder):
    data = pd.read_csv(os.path.join(root_folder, file))
    filtered_data = data.dropna(axis='columns', how='all')
    new_name = file.replace('.csv', '_modified.csv')
    filtered_data.to_csv(os.path.join(root_folder, new_name), index=False)

Basically what you are doing is looping over all file files in the directory, copying the data, doing the operations you want and then saving the data to the same location but replacing .csv with _modified.csv (to add the modified tag as you requested)

Python - importing and re-writing multiple CSV files

1 Answers1