Reducing memory usage when indexing in Numpy

Asked Nov 03 '15 at 13:05

Active Nov 03 '15 at 15:45

Viewed 1,065 times

I have a large Numpy matrix act with dtype=np.float32 and two vectors of the same length, raw_id and raw_label. I want to sort all 3 objects based on the values in raw_id. However, I get a memory error when running this script. I've isolated it to act[sortind,:] in the function below. How can I optimize the memory usage?

The arrray act is roughly 1400000 x 400, whereas raw_id and raw_label is 1400000 x 1 using dtype=np.float64. It will almost fit into my 12gb of memory along with the remaining variables that I have initialised.

def sort_by_id(act, raw_id, raw_label):
    sortind = np.argsort(raw_id)
    return act[sortind,:], raw_id[sortind], raw_label[sortind]

# calling function with same variables
act, raw_id, raw_label = sort_by_id(act, raw_id, raw_label)

edited Nov 03 '15 at 15:45

asked Nov 03 '15 at 13:05

pir

5,513
12
63
101

This is probably because [fancy indexing always returns a copy rather than a view](http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing). – ali_m Nov 03 '15 at 14:23
Related: http://stackoverflow.com/q/26239802/1461210 – ali_m Nov 03 '15 at 14:24
Also: http://stackoverflow.com/q/31359980/1461210 – ali_m Nov 03 '15 at 14:56
How big is your array? How much RAM do you have? – ali_m Nov 03 '15 at 14:58
Thanks for the related posts - I'm looking into them. See my update regarding the size and RAM. – pir Nov 03 '15 at 15:02
A 1400000 * 400 array of 32bit floats is only ~2.24 GB. What are `raw_id` and `raw_label` like? – ali_m Nov 03 '15 at 15:36
See my update. That sounds strange.. I'll investigate further. – pir Nov 03 '15 at 15:46

Reducing memory usage when indexing in Numpy

0 Answers0