How can I create a numpy .npy file in place on disk?

Question

Is it possible to create an .npy file without allocating the corresponding array in memory first?

I need to create and work with a large numpy array, too big to create in memory. Numpy supports memory mapping, but as far as I can see my options are either:

Create a memmapped file using numpy.memmap. This creates the file directly on disk without allocating memory, but doesn't store the metadata, so when I re-map the file later I need to know its dtype, shape, etc. In the following, notice that not specifying the shape results in the memmap being interpreted as flat array:
```
In [77]: x=memmap('/tmp/x', int, 'w+', shape=(3,3))


In [78]: x
Out[78]: 
memmap([[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]])


In [79]: y=memmap('/tmp/x', int, 'r')


In [80]: y
Out[80]: memmap([0, 0, 0, 0, 0, 0, 0, 0, 0])
```
Create an array in memory, save it using numpy.save, after which it can be loaded in memmapped mode. This records metadata with the array data on disk, but requires that memory be allocated for the entire array at least once.

Why not just write the meta-data to file as well? – Chinmay Kanchi Dec 02 '10 at 15:06 — Chinmay Kanchi, Dec 02 '10 at 15:06

score 13 · Accepted Answer · answered Jan 06 '11 at 22:02

I had the same question and was disappointed when I read Sven's reply. Seems as though numpy would be missing out on some key functionality if you couldn't have a huge array on file and work on little pieces of it at a time. Your case seems to be close to one of the use cases in the origional rational for making the .npy format (see: http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt).

I then ran into numpy.lib.format, which seems to be full useful goodies. I have no idea why this functionality is not available from the numpy root package. The key advantage over HDF5 is that this ships with numpy.

>>> print numpy.lib.format.open_memmap.__doc__

"""
Open a .npy file as a memory-mapped array.

This may be used to read an existing file or create a new one.

Parameters
----------
filename : str
    The name of the file on disk. This may not be a filelike object.
mode : str, optional
    The mode to open the file with. In addition to the standard file modes,
    'c' is also accepted to mean "copy on write". See `numpy.memmap` for
    the available mode strings.
dtype : dtype, optional
    The data type of the array if we are creating a new file in "write"
    mode.
shape : tuple of int, optional
    The shape of the array if we are creating a new file in "write"
    mode.
fortran_order : bool, optional
    Whether the array should be Fortran-contiguous (True) or
    C-contiguous (False) if we are creating a new file in "write" mode.
version : tuple of int (major, minor)
    If the mode is a "write" mode, then this is the version of the file
    format used to create the file.

Returns
-------
marray : numpy.memmap
    The memory-mapped array.

Raises
------
ValueError
    If the data or the mode is invalid.
IOError
    If the file is not found or cannot be opened correctly.

See Also
--------
numpy.memmap
"""

I've just got back to this to try it out. It works - thanks a lot. A word of caution: if you use the wrong mode string it might be silently accepted and a file will be created, but the header won't be written properly. The only mode string for writing a new file is 'w+'. — hamish, Mar 17 '11 at 09:19
I just used this method to write a 38G file and read (at least a few rows) out of it successfully. It is my understanding that this is not supposed to work bc there is a 2GB limit on this files. Any ideas if this limit is still true or how to observe whatever problem might happen when dealing with large files? — mathtick, May 16 '13 at 16:49
@hamish Probably the only correct way to open it for writing is 'w+' because the header needs to be updated AFTER appending data: the array size (which is written in the shape field in the header) is not known when you create the file. — Enrico Detoma, Nov 24 '14 at 09:52
its also really important that the shape is a tuple of int's anything else that usually works with arrays causes errors when trying to load it at a later time. — gauteh, Aug 16 '17 at 10:57

Sven Marnach · Answer 2 · 2010-12-02T14:33:05.047

4

As you have found out yourself, NumPy is mainly targetted at handling data in memory. There are different libraries for handling data on disk, the one most commonly used today probably being HDF5. I suggest having a look at h5py, an excellent Python wrapper for the HDF5 libraries. It is designed to be used together with NumPy, and its interface is easy to learn if you already know NumPy. To get an impression how it tackles your problem, read the documentation of Datasets.

For the sake of completeness I should mention PyTables, which seems to be the "standard" way of handling large datasets in Python. I did not use it because h5py appealed more to me. Both libraries have FAQ entries defining their scope against the other one.

edited Dec 02 '10 at 14:33

answered Dec 02 '10 at 13:35

Sven Marnach

574,206
118
941
841

Aha. Thanks muchly. I've come across those in my browsings before, but was suffering from a mental block ... – hamish Dec 02 '10 at 14:16
Indeed, the interface of h5py is really easy to learn if you already know Numpy. Have just learned to use it! Thanks! – mcrisc Nov 09 '16 at 11:55

How can I create a numpy .npy file in place on disk?

2 Answers2

Linked