How to overcome MemoryError of numpy.unique

Question

I am using Numpy version 1.11.1 and have to deal with an two-dimensional array of

my_arr.shape = (25000, 25000)

All values are integer, and I need a unique list of the arrays values. When using lst = np.unique(my_arr) I am getting:

Traceback (most recent call last):
  File "<pyshell#38>", line 1, in <module>
    palette = np.unique(arr)
  File "c:\Python27\lib\site-packages\numpy\lib\arraysetops.py", line 176, in unique
    ar = np.asanyarray(ar).flatten()
MemoryError

My machine has only 8 GB RAM, but I tried it with another machine with 16 GB RAM, and the result is the same. Monitoring the memory and CPU usage doesn't show that the problems are related to RAM or CPU.

In principle, I know the values the array consists of, but what if the input changes... Also, if I want to replace values of the array by another (let's say all 2 by 0), will it need a lot of RAM as well?

Sounds like your problem right there. Don't use 32 bit python. Also, try using as small an int as possible; perhaps a single byte per value will do? — Eelco Hoogendoorn, Oct 05 '16 at 15:29
Well, the change to 64 bit I can do. All arrays at hand only consist out of 0, 1, or 2 - just the shape is huge. — TomGeo, Oct 05 '16 at 15:32
Switching to 64 bit will allow you to use more than 1gb per process. Using my_arr.astype(np.int8) will gain you another factor 4 in memory savings. — Eelco Hoogendoorn, Oct 05 '16 at 15:46
Did you ever figure this out? Did using 64-bit work for you, or did you have to come up with something else? — jpmc26, May 12 '18 at 06:29

score 0 · Answer 1 · edited Oct 05 '16 at 16:12

0

Python 32-bit can't access more than 4 GiB RAM (often ~2.5 GiB). The obvious answer would be to use the 64-bit version. If that doesn't work, another solution would be to use numpy.memmap and memory-map the array into a file stored on disk.

edited Oct 05 '16 at 16:12

smerlin

6,446
3
35
58

answered Oct 05 '16 at 15:32

fzzle

1,466
5
23
28

2

I still got a `MemoryError` with 64-bit and using `numpy.memmap`. The array is `float`s with shape `(8465, 103114)`, only slightly bigger than the OP's array. (It comes from a geographic raster.) – jpmc26 May 12 '18 at 06:26

How to overcome MemoryError of numpy.unique

1 Answers1

Linked