How to load all .txt files in a folder with separate names - Python

Question

I have to create a bunch of numpy arrays by importing all the .txt files from a folder. This is the way I'm doing it right now:

wil_davide_noIA_IST_nz300           = np.genfromtxt("%s/davide/wil_davide_noIA_IST_nz300.txt" %path)
wig_davide_multiBinBias_IST_nz300   = np.genfromtxt("%s/davide/wig_davide_multiBinBias_IST_nz300.txt" %path)
wig_davide_multiBinBias_PySSC_nz300 = np.genfromtxt("%s/davide/wig_davide_multiBinBias_PySSC_nz300.txt" %path)
wig_davide_noBias_PySSC_nz300       = np.genfromtxt("%s/davide/wig_davide_noBias_PySSC_nz300.txt" %path)
wig_davide_IST_nz300                = np.genfromtxt("%s/davide/wig_davide_IST_nz300.txt" %path)
wig_davide_noBias_IST_nz300         = np.genfromtxt("%s/davide/wig_davide_noBias_IST_nz300.txt" %path)
wig_davide_PySSC_nz300              = np.genfromtxt("%s/davide/wig_davide_PySSC_nz300.txt" %path)
...

from the folder

Can I automate the process somehow? Note that I'd like the array to have the same name as the imported file (without the .txt, of course).
Thank you very much

Is this something that can help? https://stackoverflow.com/questions/18262293/how-to-open-every-file-in-a-folder — kabooya, Mar 11 '21 at 14:06
Although I would not recommend it if there are any security concerns with the files or file names you can use `exec` it executes a string as python code. Combined with the `os.listdir` function you can write your python code in a string an then execute it. The variables are available after that. — Camaendir, Mar 11 '21 at 14:10
Why do you want to automate it? Once you start using many variables and names like that, you are stuck with using them through out your code. In Python you can store arrays like that in lists or dicts. You don't need to assign each to a separate variable. — hpaulj, Mar 11 '21 at 19:15
@hpaulj Thanks, could you expound a bit on that? I know the basic working of dictionaries, but how would you use one in this case? — scott, Mar 19 '21 at 10:06

prkmk · Answer 1 · 2021-03-11T14:15:07.673

This is how you can get all the filenames.

path is whatever is your file path ("./" if in the same directory)

import os

path = "./"

filenames = []
for filename in os.listdir(path):
  filenames.append[file]

...

Also, you can filter some files with the if-structure provided. Here I'm selecting only txt files.

filter is some filter you can apply (use "." not in file to get directories)

import os

path = "./"
filter = ".txt"

txt_filenames = []
for filename in os.listdir(path):
  if filter in file:
    txt_filenames.append[filename]

...

score 1 · Answer 2 · answered Mar 11 '21 at 14:18

A heads up: You should not do this, if there are any security concerns with the files or their names.

Otherwise you can use the exec function to get the variables you want. For a list approach see the other solution.

import os
path_to_folder = "."
command = ""
for f in os.listdir(path_to_folder):
    if f.endswith(".txt"):
        command += f.replace(".txt", "") + f" = np.genfromtxt('{path_to_folder}/{f}')\n"
exec(command)

Depended on your folder/file structure you would need to change up the code a bit, but that should work like asked.

Paul M. · Accepted Answer · 2021-03-11T14:30:36.233

I haven't tested this, but I think this pseudo-code should work - the only thing "pseudo" about it is the hardcoded "dir/to/files" string, which you'll have to change to the path to the directory containing the text files. In modern Python you would use the pathlib or glob standard library modules to iterate over all text files in a given directory. Creating a variable number of variables, with variable names determined at runtime is never a good idea. Instead, keep your numpy arrays in some kind of collection. In this case, I would suggest a dictionary, so that you can access the individual numpy arrays by key, where the key is the name of the corresponding file as a string - without the extension of course:

def get_kv_pairs():
    from pathlib import Path
    for path in Path("dir/to/files").glob("*.txt"):
        yield path.stem, np.genfromtxt(str(path))
        

arrays = dict(get_kv_pairs())

print(arrays["wil_davide_noIA_IST_nz300"]) # Here's how you would access the individual numpy arrays.

How to load all .txt files in a folder with separate names - Python

3 Answers3