8

Is there a method to save a numpy memmap array into a .npy file? Apparently, there is a method to load such an array from a .npy file as follows

data = numpy.load("input.npy", mmap_mode='r')

but flushing the file is not equivalent to storing it in a .npy format.

If flushing is the only way to go then is there a way to infer the shape of the stored array? I would prefer to have dynamic shape which is automatically stored and retrieved (possibly as memmap again) in another script.

I have searched on various places about this but didn't find get any result. I way to store into .npy I do now is

numpy.save(output.filename, output.copy())

which defeats the idea of using memmap but preserves the shape.

NOTE: I know about hdf5 and h5py but I was wondering if there is a pure numpy solution to this.

pevogam
  • 536
  • 1
  • 6
  • 15

2 Answers2

11

is there a way to infer the shape of the stored array?

No. As far as np.memmap is concerned the file is just a buffer - it stores the contents of the array, but not the dimensions, dtype etc. There's no way to infer that information unless it's somehow contained within the array itself. If you've already created an np.memmap backed by a simple binary file then you would need to write its contents to a new .npy file on disk.

You could avoid generating a copy in memory by opening the new .npy file as another memory-mapped array using numpy.lib.format.open_memmap:

import numpy as np
from numpy.lib.format import open_memmap

# a 10GB memory-mapped array
x = np.memmap('/tmp/x.mm', mode='w+', dtype=np.ubyte, shape=(int(1E10),))

# create a memory-mapped .npy file with the same dimensions and dtype
y = open_memmap('/tmp/y.npy', mode='w+', dtype=x.dtype, shape=x.shape)

# copy the array contents
y[:] = x[:]
Community
  • 1
  • 1
ali_m
  • 71,714
  • 23
  • 223
  • 298
  • 1
    This `open_memmap` function is a great finding - I just needed the way to start a `.npy` backed array but adding an option to save an array that could be stuck in a binary file is even better. – pevogam Apr 25 '16 at 01:07
  • open_memmap is really great finding – zhao yufei May 30 '19 at 02:02
5

An array saved with np.save is essentially a memmap with a header specifying dtype, shape, and element order. You can read more about it in the numpy documentation.

When you create your np.memmap, you can reserve space for that header with the offset parameter. The numpy documentation specifies that the header length should be a multiple of 64:

Let's say you reserve 2 * 64 = 128 bytes for the header (more on this below):

import numpy as np
x = np.memmap('/tmp/x.npy', mode='w+', dtype=np.ubyte, 
              shape=(int(1E10),), offset=128)

Then, when you are finished manipulating the memmap, you create and write the header, using np.lib.format:

header = np.lib.format.header_data_from_array_1_0(x)

with open('/tmp/x.npy', 'r+b') as f:
    np.lib.format.write_array_header_1_0(f, header)

Note that this writes the header from the start of the memmap file, so if len(header) > 128, then it will overwrite part of the data, and your file will not be readable. The header is a fixed length magic string (6 bytes), two version bytes, two bytes specifying the header length, and a string representation of a dictionary specifying 'shape', 'descr', and 'order'. If you know the shape and the dtype (descr) of your array, you can easily compute the header length (I fixed it at 128 above, for the sake of simplicity).

After writing the header you can load the data using np.load:

y = np.load('/tmp/x.npy')

If the memmap you saved is large you might want to load the data as a memmap again:

y = np.load('/tmp/x.npy', mmap_mode='r')
maarten
  • 412
  • 5
  • 7