How to dynamically create variables from .csv files in a folder?

Question

I have a folder with any number of .csv files. For all intents and purposes let's say I have three now: map1.csv, map2.csv and map3.csv.

I want to import the contents of these .csv files into a dataframe with a name that is the same as the filename they came from. So the content of map1.csv in folder ### needs to be in a dataframe named map1 (or map1.csv).

I know creating variables dynamically is not the best thing, but it is what I need exactly. So if there are better ways to tackle my problem I'd be happy to learn.

Thanks in advance :)

Here's what I tried: filenames obviously contains the names of the files csv_files contains the paths to those files count is number of files in the folder path is the path of the folder which contains the files

import pandas as pd
from tkinter.filedialog import askdirectory
import fnmatch 
import os
import glob

path = askdirectory(title='Select Folder') # shows dialog box and return the path
count = int(len(fnmatch.filter(os.listdir(path), '*.*')))
filenames = os.listdir(path)
csv_files = glob.glob(os.path.join(path, "*.csv"))

for i in range(count):
    var_name = filenames[i]
    value = pd.read_csv(csv_files[i])
    globals()[var_name]= value

print(Map1.csv)

How about using a dictionary, where the keys are filenames and the values are your dataframes? https://stackoverflow.com/questions/1373164/how-do-i-create-variable-variables. (By the way, `map1.csv` is an invalid variable name because of the period, but it is a valid dictionary key.) — slothrop, May 02 '23 at 08:20
I've have seen more posts indeed about dictionaries, however I am unfamiliar with dictionaries sadly. In your example given, would it be possible to loop through an n number of files using a dict like I need? And if so, what would that look like in python. — Carst Hoopman, May 02 '23 at 08:33

slothrop · Answer 1 · 2023-05-02T09:00:52.333

A Python dictionary is the appropriate data structure here: the keys would be your filenames and the values the dataframes you create.

The code could also be improved by removing count and filenames (which include any non-CSV files that might be present) and just working with csv_files.

For example, starting from the point where you already have csv_files:

df_dict = {}  # Create an empty dictionary

for csv_f in csv_files:
    fname = os.path.basename(csv_f)  # Just the filename without directory prefix
    df = pd.read_csv(csv_f)
    df_dict[fname] = df  # Add a dictionary entry where the key is <fname> and the value is <df>

print(df_dict['Map1.csv'])  # Get the value corresponding to a particular dictionary key

It's definitely worth reading more and getting familiar with dictionaries, since they're a fundamental data structure in Python. You could look at: https://realpython.com/python-dicts/

How to dynamically create variables from .csv files in a folder?

1 Answers1