Making a Python script to load all npy/npz files in a directory

Question

I want to load all npy/npz files into my interactive python shell, so I don't have to go:

var1 = np.load('var1.npy')

for each one. I made this script, but it doesn't work because it the variables namespace is just within the script (pretend the indenting is correct). What is the right way to do this?

def load_all():
import numpy as np
from os import listdir
from os.path import isfile, join
from os import getcwd

mypath = getcwd()
print 'loading all .npy and .npz files from ',mypath
files = [ f for f in listdir(mypath) if isfile(join(mypath,f)) ]

for f in files:  
    if f[-4:] in ('.npy','.npz'):
        name = f[:-4]+'_'+f[-3:]
        print 'loading', f, 'as', name
        var = np.load(f)
        exec(name + " = var")

Peter DeGlopper · Answer 1 · 2015-02-04T23:37:36.797

3

I'd use glob. For example, glob.glob('*.np[yz]') will return a list of all .npy and .npz filenames in the current directory. You can then iterate over that list, loading each filename in turn. Are you trying to then put the results of loading them into local variables that match the filename? There are safer designs than that - I'd use a single dictionary and use the names as keys, something like:

numpy_vars = {}
for np_name in glob.glob('*.np[yz]'):
    numpy_vars[np_name] = np.load(np_name)

edited Feb 04 '15 at 23:37

answered Jan 25 '14 at 20:07

Peter DeGlopper

36,326
7
90
83

"Are you trying to then put the results of loading them into local variables that match the filename?" - YES! I've seen a lot of similar questions on SO and everyone always says "use a dictionary" even though that's not what the OP asked for. So... 1. Why? and 2. How to do it without a dictionary? (or maybe I can just get the variables out of the dictionary with one of the methods here: http://stackoverflow.com/questions/4650622/how-can-i-load-all-keys-from-a-dict-as-local-variables-a-better-aproach) – capybaralet Jan 25 '14 at 20:22
Oh yeah, wait, so will this work if I wrap it in a script? I'm also wondering about that aspect of it. I mean, I could put this in a script and then return the dictionary, but then if I want to pull the names out of the dictionary and load them, I'd still have to do that manually, right? I want a script that I can run that will just load everything and assign all the variables to the loaded arrays. – capybaralet Jan 25 '14 at 20:27
Everyone always says to use a dictionary because that's the best practice in Python if you don't know beforehand what your variable names will be. Injecting arbitrary names into the local namespace is slow, unsafe or both. – Peter DeGlopper Jan 25 '14 at 22:34
I'm writing code for personal use on research projects. AFAIK, "injecting arbitrary names into the local namespace" raises no speed or safety issues for my use case. – capybaralet Nov 26 '15 at 07:12

score 0 · Answer 2 · answered Aug 28 '16 at 15:32

Simple and pythonic.

import numpy as np
from os import listdir

directory_path = '.'
file_types = ['npy', 'npz']

np_vars = {dir_content: np.load(dir_content)
           for dir_content in listdir(directory_path)
           if dir_content.split('.')[-1] in file_types}

Making a Python script to load all npy/npz files in a directory

2 Answers2