2

So I am doing hyperspectral image classification atm (python + tensorflow) and with one specific test i ran into problem that i don't have enough RAM.

General idea: i have hyperspectral image (ndarray, dtype = float64) WxHxD i need to extract smaller cubes/windows from it for classified pixels WSxWSxD, but in order to feed them into ANN i need to reshape them into SxWSxWsxDx1, where S is samples and 1 is for channels and that's where i don't have enough memory.

as it is right now in my code i extract all classified pixels into similarly shaped containers per class, which fails because given window size WS = 27, depth D = 103 and there are around 42k classified pixels.

42776 * 27 * 27 *103 * 8 / 1024^2 = 24505 MB

total of 24.5 GB which i don't have.

I looked into whether i can concatenate numpy views/slices such that it would still be a sort of view, but Here people say that i can't do that, because it wouldn't be contiguous which is fair enough.

So the only option that i am seeing right now is to extract cubes as python list filled with views/slices and then have a buffer that would be refilled and fed into ANN, which would be a a bit annoying because that's probably only test with such big window+depth.

My question is whether there is some other better way to force numpy stack/concat as a view-like object without inflating memory (or maybe there is some special sacred technology that am not aware of that can help) or is "buffer + feeding in portions" is the most rational solution?

deni0077
  • 21
  • 1
  • did you try to switch off eager mode? – Marat Oct 14 '20 at 02:30
  • @Marat with this test, i didn't even reach tensorflow part of the code. Numpy just fails to arrange samples in the memory as independent units and quits the script. – deni0077 Oct 14 '20 at 02:51
  • 2
    In `numpy` we can get moving windows and other regular slices (overlapping or not) using `np.lib.stride_tricks.as_strided`. This uses `shape` and `strides` to construct a `view` of the source array. This can be useful for calculating things like moving averages, but it will produce your kind of inflation if we try to reshape or `copy` it. – hpaulj Oct 14 '20 at 03:25
  • Can't you pass these views to the batched tensorflow object without first concatenating them with `numpy`? – hpaulj Oct 14 '20 at 03:28
  • @hpaulj can you elaborate on this tensorflow batched object? Not sure what you mean by it. – deni0077 Oct 14 '20 at 11:57

1 Answers1

0

You can use np.empty, for example

one = np.random.rand(5,4,1,2,3)
two = np.random.rand(5,4,1,2,3)

vstack_shape = list(one.shape)
vstack_shape[0] *= 2

stacked = np.empty(vstack_shape)
stacked[0:5,...] = one
stacked[5:,...] = two

np.all(np.vstack((one,two)) - stacked == 0.)

instead of directly applying np.vstack, you can create a placeholder using np.empty in the first place, then the arrays one and two can be loaded (for example using np.load) directly to the corresponding positions in the placeholder.

meTchaikovsky
  • 7,478
  • 2
  • 15
  • 34
  • wouldn't np,empty still claim RAM space on array creation? As far as I understand in this example stacked variable doesn't share memory with original variables which would again inflate memory. – deni0077 Oct 14 '20 at 02:47
  • @deni0077 If the reshaped array is still larger than RAM, then this solution will also inflate memory. However, if the reshaped array can reside in RAM and the overhead is caused by operations like `stack` (because copies of the arrays to stack will be created), then you can use `np.empty`, claim the memory in the first place, then load the arrays one by one. – meTchaikovsky Oct 14 '20 at 02:53