Generator and reading files optimization

Question

I have thousands of binary files that I have to read and store in memory to work on the data. I already have a function that permit to read those data, but I would like to improve it, because it is kind of slow.

The data are organize this way :

• 1000 cubes.
• each cube is written in 10 binary files.

For the moment I have a reading function that can read and return ONE cube in a numpy array (read_1_cube). Then I loop over all the file to extract all the cube and I concatenate them.

def read_1_cube( dataNum ):
    ### read the 10 subfiles and concatenate arrays
    N_subfiles = 10
    fames_subfiles = ( '%d_%d'%(dataNum,k) for k in range(N_subfiles) )
    return np.concatenate( [np.fromfile( open(fn,'rb'), dtype=float, count=N*N*N ).reshape((N,N,N)) for fn in fames_subfiles], axis=2 )

TotDataNum = 1000
my_full_data = np.concatenate( [read_1_cube( d ) for d in range( TotDataNum )], axis=0 )

I try to work with generators to limit the amount of memory used. With those function it took ~2.5s per file, so 45min hour for the 1000 file, in the end I will have 10000 files, so it is not doable (of course, I will not read the 10000 files a t onces, but steel I can not work if it take 1h for 1000 files).

My questions:

• do you know a way to optimize the read_1_cube and the generation of my_full_data ?
• your do see a better way (without the read_1_cube) ?
• An other optimization way: do you know if there is a concatenate function that can work on a generator (like sum(), min(), max(), list()... )?

---

Edit: Following the comment of @liborm about np.concatenate I find other equivalent functions (stack concatenate question): np.r_, np.stack, np.hstack. The good point is that stack can take a generator in input. So I push as far as possible with generator, to create the actual data array only at the end.

def read_1_cube( dataNum ):
    ### read the 10 subfiles and retur cube generator
    N_subfiles = 10
    fames_subfiles = ( '%d_%d'%(dataNum,k) for k in range(N_subfiles) )
    return (np.fromfile( open(fn,'rb'), dtype=float, count=N*N*N ).reshape((N,N,N)) for fn in fames_subfiles)

def read_N_cube( datanum ):
    ### make a generator of 'cube generator'
    C = ( np.stack( read_1_cube( d ), axis=2 ).reshape((N,N,N*10)) for d in range(datanum) )
    return np.stack( C ).reshape( (datanum*N,N,N*N_subfiles) )

### The full allocation is done here, just once
my_full_data = read_N_cube( datanum )

It is quicker than the first version, where the first version needed 2.4s to read 1 file, the second take 6.2 to read 10 files!

I think that there are not so much place for optimization, but I am sure that there is still a better algorithm out there!

Answer 1

To get a good performance (generally) you want to allocate as little as possible - this should allocate only the big array beforehand, and then each of the small ones during read. Using stack or concatenate will probably (re)alloc memory and copy data around...

I don't have the data to test it, consider this rather a 'pseudocode':

def read_one(d, i):
    fn = '%d_%d' % (d, i)
    return np.fromfile(open(fn,'rb'), dtype=float, count=N*N*N).reshape((N,N,N))

res = np.zeros((N * TotDataNum, N, N * N_subfiles))
for dat in range(TotDataNum):
    ax0 = N * dat
    for idx in range(N_subfiles):
        ax2 = N * idx
        res[ax0:ax0+N, :, ax2:ax2+N] = read_one(dat, idx)

Answer 2

I haven't fully worked out why your change has sped things up, but I doubt if it's the use of stack.

To test this:

In [30]: N=50
In [31]: arr = np.arange(N*N).reshape(N,N)
In [32]: np.stack((arr for _ in range(N*10))).shape
Out[32]: (500, 50, 50)
In [33]: np.concatenate([arr for _ in range(N*10)]).shape
Out[33]: (25000, 50)

times:

In [34]: timeit np.stack((arr for _ in range(N*10))).shape
100 loops, best of 3: 2.45 ms per loop
In [35]: timeit np.stack([arr for _ in range(N*10)]).shape
100 loops, best of 3: 2.43 ms per loop
In [36]: timeit np.concatenate([arr for _ in range(N*10)]).shape
1000 loops, best of 3: 1.56 ms per loop

The use of generator comprehension in stack has no advantage. concatenate doesn't take a generator, but is faster.

np.stack is a just a convenience cover for concatenate. Its code is:

arrays = [asanyarray(arr) for arr in arrays]
...
expanded_arrays = [arr[sl] for arr in arrays]
return _nx.concatenate(expanded_arrays, axis=axis)

It does 2 list comprehensions on the input argument, once to make sure they are arrays, and again to add a dimension. That explains why it accepts an generator list, and why it is slower.

concatenate is compiled numpy code, and expects a 'sequence':

TypeError: The first input argument needs to be a sequence