How to merge very large numpy arrays?

Question

I will have many Numpy arrays stored in npz files, which are being saved using savez_compressed function.

I am splitting the information in many arrays because, if not, the functions I am using crash due to memory issues. The data is not sparse.

I will need to joint all that info in one unique array (to be able to process it with some routines), and store it into disk (to process it many times with diffente parameters).

Arrays won't fit into RAM+swap memory.

How to merge them into an unique array and save it to a disk?

I suspect that I should use mmap_mode, but I do not realize exactly how. Also, I imagine that can be some performance issues if I do not reserve contiguous disk space at first.

I have read this post but I still cannot realize how to do it.

---

EDIT

Clarification: I have made many functions to process similar data, some of them require an array as argument. In some cases I could pass them only part of this large array by using slicing. But it is still important to have all the info. in such an array.

This is because of the following: The arrays contain information (from physical simulations) time ordered. Among the argument of the functions, the user can set the initial and last time to process. Also, he/she can set the size of the processing chunk (which is important because this affect to the performance but allowed chunk size depend on the computational resources). Because of this, I cannot store the data as separated chunks.

The way in which this particular array (the one I am trying to create) is built is not important while it works.

Answer 1

This would be an example how to write a 90GB of easily compressible data to disk. The most important points are mentioned here https://stackoverflow.com/a/48405220/4045774

The write/read speed should be in the range of (300 MB/s,500MB/s) on a nomal HDD.

Example

import numpy as np
import tables #register blosc
import h5py as h5
import h5py_cache as h5c
import time

def read_the_arrays():
  #Easily compressable data
  #A lot smaller than your actual array, I do not have that much RAM
  return np.arange(10*int(15E3)).reshape(10,int(15E3))

def writing(hdf5_path):
  # As we are writing whole chunks here this isn't realy needed,
  # if you forget to set a large enough chunk-cache-size when not writing or reading 
  # whole chunks, the performance will be extremely bad. (chunks can only be read or written as a whole)
  f = h5c.File(hdf5_path, 'w',chunk_cache_mem_size=1024**2*1000) #1000 MB cache size
  dset = f.create_dataset("your_data", shape=(int(15E5),int(15E3)),dtype=np.float32,chunks=(10000,100),compression=32001,compression_opts=(0, 0, 0, 0, 9, 1, 1), shuffle=False)

  #Lets write to the dataset
  for i in range(0,int(15E5),10):
    dset[i:i+10,:]=read_the_arrays()

  f.close()

def reading(hdf5_path):
  f = h5c.File(hdf5_path, 'r',chunk_cache_mem_size=1024**2*1000) #1000 MB cache size
  dset = f["your_data"]

  #Read chunks
  for i in range(0,int(15E3),10):
    data=np.copy(dset[:,i:i+10])
  f.close()

hdf5_path='Test.h5'
t1=time.time()
writing(hdf5_path)
print(time.time()-t1)
t1=time.time()
reading(hdf5_path)
print(time.time()-t1)

Answer 2

You should be able to load chunk by chunk on a np.memap array:

import numpy as np

data_files = ['file1.npz', 'file2.npz2', ...]

# If you do not know the final size beforehand you need to
# go through the chunks once first to check their sizes
rows = 0
cols = None
dtype = None
for data_file in data_files:
    with np.load(data_file) as data:
        chunk = data['array']
        rows += chunk.shape[0]
        cols = chunk.shape[1]
        dtype = chunk.dtype

# Once the size is know create memmap and write chunks
merged = np.memmap('merged.buffer', dtype=dtype, mode='w+', shape=(rows, cols))
idx = 0
for data_file in data_files:
    with np.load(data_file) as data:
        chunk = data['array']
        merged[idx:idx + len(chunk)] = chunk
        idx += len(chunk)

However, as pointed out in the comments working across a dimension which is not the fastest one will be very slow.