indexing with list comprehensions? Is it possible?

Question

The way I'm doing this now seems really clunky. Is there a better way to index this array?

This is the code that works:

DEM = np.empty((number_of_DEM_files, DEMfile.shape[0], DEMfile.shape[1]), 'float')

for t in range(number_of_DEM_files):
      DEM[t] = np.load(DEM_filename_list[t])

I tried doing it with list comprehensions like this:

DEM = np.load([DEM_filename_list[t] for t in range(number_of_DEM_files)])

but I get a syntax error. Is this possible? Is there even a reason to do this or is it just as slow as what I've already got? Is there a better way?

EDIT:

DEM_filename_list looks like this:

DEM_filename_list = (D:/Documents/Data/grand_canyon_2015/03-11/dem1.npy,
                     D:/Documents/Data/grand_canyon_2015/03-11/dem2.npy,
                     D:/Documents/Data/grand_canyon_2015/03-11/dem3.npy,
                     etc)

The first line creates an empty 3d array. Ultimately, I'm trying to load and store a time series of arrays in one 3d array, so that you can index it with DEM[t,i,j]

where t is the time, i is row number, and j is the column number.

unrelated but `DEM_filename_list[:number_of_DEM_files]` would do the same as your list comp — Padraic Cunningham, Jul 23 '15 at 15:58
Are you are missing an underscore in `number_of_DEM Files` ? — stellasia, Jul 23 '15 at 15:59
I don't think doing a simple `for` for this is clunky, although I didn't understand what the first line does. — lucasnadalutti, Jul 23 '15 at 16:00
What about `DEM = [np.load(f) for f in DEM_filename_list]`? Do you want to build a matrix with row values from different files? — wenzul, Jul 23 '15 at 16:01
The fundamental problem is that `np.load` does not take a list of filenames. It takes one name. But you can collect the results of the load into a list or array. — hpaulj, Jul 23 '15 at 17:36

Geeocode · Accepted Answer · 2015-07-23T16:52:56.007

1

This is a testable example, which workes fine:

import numpy as np

a = np.array((1,2))
b = np.array((3,4))

with open('12', 'wb') as f:
    np.save(f,a)
with open('34', 'wb') as f:
    np.save(f,b)    

l = DEM_filename_list

DEM = [np.load(ll) for ll in l]

print DEM

Output:

[array([1, 2]), array([3, 4])]

Or presented with your annotations:

import numpy as np

DEM = [np.load(ll) for ll in DEM_filename_list]

print DEM

Output:

 DEM_files

UPDATED:

There is no need for the row:

DEM = np.empty((number_of_DEM_files))

edited Jul 23 '15 at 16:52

answered Jul 23 '15 at 16:17

Geeocode

5,705
3
20
34

DEM is then no longer a numpy array. You might want to do something like: `DEM = np.empty((2,2)); DEM[:] = [np.load(ll) for ll in l]` instead. – Dunes Jul 23 '15 at 16:34
True, I updated, thanks. Note, that as I wrote in the update, there is no need any action before the assigning. – Geeocode Jul 23 '15 at 16:44

score 1 · Answer 2 · answered Jul 23 '15 at 16:23

I am not sure if number_of_DEM_files has a direct relation to DEM_filename_list, I assume it is one to one and DEM_filename_list is iterable (going by name), in that case I would do something like this.

DEM = np.empty((number_of_DEM_files, DEMfile.shape[0], DEMfile.shape[1])

for i, t in enumerate(DEM_filename_list):
      DEM[i] = np.load(t)

or

DEM = [np.load(t) for t in DEM_filename_list]

Ramón J Romero y Vigil · Answer 3 · 2015-07-23T16:27:19.987

-1

You can use the enumerate built-in function do get an enumerate object of (index, value) and use the index in the resulting pair.

If performance is an issue you can use the numpy enumerate function: ndenumerate.

edited Jul 23 '15 at 16:27

answered Jul 23 '15 at 16:05

Ramón J Romero y Vigil

17,373
7
77
125

Don't do this in numpy. It'll cause you to lose all of the performance gain you get from vectorization. – Slater Victoroff Jul 23 '15 at 16:16
You get similar performance from range as from enumerate: http://stackoverflow.com/questions/11990105/rangelenlist-or-enumeratelist – Ramón J Romero y Vigil Jul 23 '15 at 16:21

indexing with list comprehensions? Is it possible?

3 Answers3