How to reshape a big array (Memory Error)

Question

I have a very big 6D array as (225, 97, 225, 32, 32, 32). I want to reshape it into 4D array like (225*97*225, 32, 32, 32). I tried to used python 2.7 in ubuntu 14.04 with bellow code but I got the memory error. How could I solve it? Thanks

import numpy as np

#input_6D shape (225, 97, 225, 32, 32, 32)
input_4D= input_6D.reshape(input_6D.shape[0]*input_6D.shape[1]*input_6D.shape[2], input_6D[0],input_6D[1],input_6D[2])

The error is

input_4D= input_6D.reshape(input_6D.shape[0]*input_6D.shape[1]*input_6D.shape[2], input_6D[0],input_6D[1],input_6D[2]) MemoryError

This is step what I did. First, I load a 3D input with shape 256x128x256. Then I used the code bellow to got 6D array as (225, 97, 225, 32, 32, 32) where patch shape is 32, 32, 32

def patch_extract_3D(input,patch_shape):
    patches_6D = np.lib.stride_tricks.as_strided(input, ((input.shape[0] - patch_shape[0] + 1) // xstep, (input.shape[1] - patch_shape[1] + 1) // ystep,
                                                  (input.shape[2] - patch_shape[2] + 1) // zstep, patch_shape[0], patch_shape[1], patch_shape[2]),
                                                  (input.strides[0] * xstep, input.strides[1] * ystep,input.strides[2] * zstep, input.strides[0], input.strides[1],input.strides[2]))

With this huge amount of data, you'll probably have to dump it to disk (file or database) first, and read it again. — Eric Duminil, Feb 25 '17 at 14:27
Have you provide any solution by code? Sorry I did not catch full your solution — user3051460, Feb 25 '17 at 14:28
Are you sure you're running the same command you show in the question? Because to my eye it looks like you won't get as far as MemoryError with that command. You should get something like "only length-1 arrays can be converted to Python scalars", because you give as arguments to reshape things like `input_6D[0]`, which are not numbers but huge arrays themselves. — avysk, Feb 25 '17 at 14:28
@user3051460: My comment was just, well, a comment, not an answer. — Eric Duminil, Feb 25 '17 at 14:29
It's seems that your input is at least 400 Go. How is it possible ? — B. M., Feb 25 '17 at 14:35
`as_strided` creates a view. But almost any action on it, including this reshape, makes a copy. — hpaulj, Feb 25 '17 at 15:24
http://stackoverflow.com/a/35805797/901925 tries to explain why the reshape creates a copy. — hpaulj, Feb 25 '17 at 15:37

score 0 · Answer 1 · edited May 23 '17 at 12:25

The strided expression works with xstep,ystep,zstep=1,1,1.

if input is np.zeros((256,128,256),np.int8) with strides (32768, 256, 1), the output will have the given shape and strides (32768, 256, 1, 32768, 256, 1).

I calculate that the strides for the reshaped array will have to be (1426325504, 58982400, 262144, 1024, 32, 1).

There's no way to get that without a copy. as_strided produces a view of the original. But the reshape will make a copy that is some 200x bigger.

More on creating a copy from an as_strided at reshaping a view of a n-dimensional array without using reshape

This strided view might work with a reduction operation, such as taking the mean or sum on the last 3 dimensions. Do that first, and then try to reshape. as_strided is tricky, and doubly so when done on 3 dimensions.

How to recover 3D image from its patches in Python? - your previous question involving this striding code.

How to reshape a big array (Memory Error)

1 Answers1