Saving List of Numpy 2D arrays using numpy.save (the arrays together are jagged)

Question

I have a large image dataset. When I use the images, I have several components--a mirrored image, a regular image, an eigenvector matrix and an eigenvalue vector.

I would like to store it like:

training_sunsets_data = [cropped_training_sunsets,
                         mirrored_training_sunsets,
                         rgb_cov_eigvec_training_sunsets,
                         rgb_cov_eigval_training_sunsets]

np.save('training_sunsets_data',training_sunsets_data)

And as I was writing this I was testing it (because I was sure it would fail), and the strangest thing happened when I did this: it worked.

Further, when I loaded it back up into the code, it was type ndarray, but it is a jagged array.

How is this possible if numpy does not allow jagged multidimensional arrays? Did I just find a backdoor way to create a jagged array in numpy?

Can you post the jagged `ndarray` you got from loading the file? I am curious to see what it looks like. — gdlmx, Apr 22 '16 at 23:37
Look at `np.savez`. That saves each array by name in a file, and collects them in a `zip` archive. `np.load` handles that kind of archive. — hpaulj, Apr 22 '16 at 23:49
There's a very good answer here: https://stackoverflow.com/questions/48603119/incremently-appending-numpy-arrays-to-a-save-file — mLstudent33, Sep 04 '19 at 04:42

gdlmx · Accepted Answer · 2016-04-22T23:49:21.607

13

After testing on my machine:

  import numpy as np
  np.save('testnp.npy', [[2,3,4],[1,2]])
  np.load('testnp.npy')
  #   array([[2, 3, 4], [1, 2]], dtype=object)

As shown in the example code, the loaded object is of type ndarray, but its data type is object. That means, np.save store an array of python objects, which can be anything. According to the documentation, it seems to use python pickle to pack those objects.

So you didn't find a backdoor, it behaves just as expected.

edited Apr 22 '16 at 23:49

answered Apr 22 '16 at 23:46

gdlmx

6,479
1
21
39

...and you could as well use pickle to store sets/sequences of arrays, if you wish. – mdurant Apr 22 '16 at 23:48

score 2 · Answer 2 · answered May 28 '19 at 14:04

2

np.savez() would work in your situation. save each as a variable.

answered May 28 '19 at 14:04

Dan Erez

1,364
15
16

score 0 · Answer 3 · answered Apr 22 '16 at 23:33

So to see what you are getting at lets runs some code.

>>> a =[np.array([[1,2,3],[4,5,6]]),np.array([[1,2],[3,4]])]
>>> type(a)
<type 'list'>
>>> np.array(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: could not broadcast input array from shape (2,3) into shape (2)

We see here that we are perfectly able to make a list of np.arrays of different dimensions. We cannot however cast that list into a np.array.

I suspect based on your syntax that you are saving a list, and loading a list maintaining the type np.array for each element in the list.

Saving List of Numpy 2D arrays using numpy.save (the arrays together are jagged)

3 Answers3