1

I have a 65GB memory mapped numpy array that I want to index:

arr = [[...],[...],[...],[...]] #a 65GB memory mapped numpy array
idx = [[...],[...],[...],[...]] #indices for fancy indexing arr
idxed_arr = arr[idx]

Learning from this question, there is no easy or memory efficient way to mimic view when fancy indexing a numpy array. Tried many approaches like this, but they were either very slow or I ran out of memory.

My last resort would be to copy few samples at a time(which is very slow). I am open and willing to try anything at this point.

Community
  • 1
  • 1
Jee Seok Yoon
  • 4,716
  • 9
  • 32
  • 47
  • Why does it need to be a view? Your first link shows how to write changes back to the map. – hpaulj Jan 18 '17 at 17:29
  • @hpaulj The first link's answer copies the indexed array and makes it a view. My situation is that I am indexing large chunks from a large array. The large indexed array is so big that copying it gives me OOM. – Jee Seok Yoon Jan 23 '17 at 07:06
  • Give us more information about `idx`? – hpaulj Jan 23 '17 at 09:01
  • Array: Slide windowed MRI image (N\*W\*H\*D\*channel\*slideW\*slideH, (N=220, W=H=240, D=155, channel=4, slideW=slideH=33)), index: Index of ROIs (W\*H\*D\*C). Each index points to a window of a N, and I have about 2,000,000 windows to view for each N. – Jee Seok Yoon Jan 23 '17 at 09:30

0 Answers0