10

I am currently trying to save a list of numpy arrays into a single file, an example of such a list can be of the form below

import numpy as np
np_list = []
for i in range(10):
    if i % 2 == 0:
        np_list.append(np.random.randn(64))
    else:
        np_list.append(np.random.randn(32, 64))

I can combine all of them using into a single file using savez by iterating through list but is there any other way? I am trying to save weights returned by the function model.get_weights(), which is a list of ndarray and after retrieving the weights from the saved file I intend to load those weights into another model using model.set_weights(np_list). Therefore the format of the list must remain the same. Let me know if anyone has an elegant way of doing this.

DVK
  • 475
  • 2
  • 6
  • 27
  • the answer here seems to do the trick [NumPy save some arrays at once](https://stackoverflow.com/a/35133517/7537870) – DVK Feb 13 '20 at 02:07
  • Does this answer your question? [NumPy save some arrays at once](https://stackoverflow.com/questions/35133317/numpy-save-some-arrays-at-once) – AMC Feb 13 '20 at 02:26

1 Answers1

17

I would go with np.save and np.load because it's platform-independent, faster than savetxt and works with lists of arrays, for example:

import numpy as np

a = [
    np.arange(100),
    np.arange(200)
]
np.save('a.npy', np.array(a, dtype=object), allow_pickle=True)
b = np.load('a.npy', allow_pickle=True)

This is the documentation for np.save and np.load. And in this answer you can find a better discussion How to save and load numpy.array() data properly?

Edit

Like @AlexP mentioned numpy >= v1.24.2 does not support arrays with different sizes and types, so that's why the casting is necessary.

marcos
  • 4,473
  • 1
  • 10
  • 24
  • This does not work for a list of numpy arrays – DVK Feb 13 '20 at 02:09
  • @DVK you can set `allow_pickle=True` and it works. – marcos Feb 13 '20 at 02:11
  • 4
    Careful with this approach. `save` will do `np.array(a)` before saving. So `b` will be an array, not a list. And as others have found out, making an array from a list of arrays has its hidden dangers. Don't do it naively. – hpaulj Feb 13 '20 at 02:27
  • 1
    @hpaulj oh, that's True! Thanks for the insight. – marcos Feb 13 '20 at 02:28
  • 1
    It does not let me answer, so I go here. It did not worked for me. The solution was to use `np.savez`. I would do the complete answer if I could comment. – J Agustin Barrachina Jul 01 '22 at 13:49
  • 1
    using numpy 1.24.2 and it no longer works if storing arrays of different size or different type. Use instead: np.save('a.npy', np.array(a,dtype=object), allow_pickle=True) – AlexP Feb 09 '23 at 02:29
  • @AlexP that's true. I'll update the answer. – marcos Feb 09 '23 at 18:37