73

I am trying to implement algorithms for 1000-dimensional data with 200k+ datapoints in python. I want to use numpy, scipy, sklearn, networkx, and other useful libraries. I want to perform operations such as pairwise distance between all of the points and do clustering on all of the points. I have implemented working algorithms that perform what I want with reasonable complexity but when I try to scale them to all of my data I run out of RAM. Of course, I do, creating the matrix for pairwise distances on 200k+ data takes a lot of memory.

Here comes the catch: I would really like to do this on crappy computers with low amounts of RAM.

Is there a feasible way for me to make this work without the constraints of low RAM? That it will take a much longer time is really not a problem, as long as the time reqs don't go to infinity!

I would like to be able to put my algorithms to work and then come back an hour or five later and not have it stuck because it ran out of RAM! I would like to implement this in python, and be able to use the numpy, scipy, sklearn, and networkx libraries. I would like to be able to calculate the pairwise distance to all my points etc

Is this feasible? And how would I go about it, what can I start to read up on?

Anton Menshov
  • 2,266
  • 14
  • 34
  • 55
Ekgren
  • 1,024
  • 1
  • 9
  • 13
  • 5
    I want to be able to perform, for example, pairwise distance between all points in a 200.000 x 1000 matrix in python without having enough ram to keep the whole distance matrix in memory. I am looking for information on how to do that :) so more concrete answers then a vague "look into two whole sub fields of computer science" would be helpfull! – Ekgren Apr 22 '13 at 15:26
  • 8
    You probably want to take a look at numpy's [memmap](http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html) and possibly [PyTables](http://www.pytables.org) as a starting point. – Henry Gomersall Apr 22 '13 at 15:42
  • From the first related question below the user @cronos suggested to [use `h5py`](http://www.h5py.org/docs/intro/quick.html#quick), and I believe it can be used for your problem too. 1-[Is it possible to np.concatenate memory-mapped files?](http://stackoverflow.com/questions/13780907/is-it-possible-to-np-concatenate-memory-mapped-files) 2-[Concatenate Numpy arrays without copying](http://stackoverflow.com/questions/7869095/concatenate-numpy-arrays-without-copying) – Saullo G. P. Castro May 02 '13 at 17:03

1 Answers1

98

Using numpy.memmap you create arrays directly mapped into a file:

import numpy
a = numpy.memmap('test.mymemmap', dtype='float32', mode='w+', shape=(200000,1000))
# here you will see a 762MB file created in your working directory

You can treat it as a conventional array: a += 1000.

It is possible even to assign more arrays to the same file, controlling it from mutually sources if needed. But I've experiences some tricky things here. To open the full array you have to "close" the previous one first, using del:

del a    
b = numpy.memmap('test.mymemmap', dtype='float32', mode='r+', shape=(200000,1000))

But openning only some part of the array makes it possible to achieve the simultaneous control:

b = numpy.memmap('test.mymemmap', dtype='float32', mode='r+', shape=(2,1000))
b[1,5] = 123456.
print a[1,5]
#123456.0

Great! a was changed together with b. And the changes are already written on disk.

The other important thing worth commenting is the offset. Suppose you want to take not the first 2 lines in b, but lines 150000 and 150001.

b = numpy.memmap('test.mymemmap', dtype='float32', mode='r+', shape=(2,1000),
                 offset=150000*1000*32/8)
b[1,2] = 999999.
print a[150001,2]
#999999.0

Now you can access and update any part of the array in simultaneous operations. Note the byte-size going in the offset calculation. So for a 'float64' this example would be 150000*1000*64/8.

Other references:

Community
  • 1
  • 1
Saullo G. P. Castro
  • 56,802
  • 26
  • 179
  • 234
  • 2
    Sorry I didn't understand what you've done. Have you created a file using 'w+' called 'test.mymemmap' which you've stored into the memory by assigning the variable 'a'. But then you've deleted it, and then read the file using 'r+' and stored in the variable 'b'. I'm not sure of what you've deed. I have a large file called 'myfile.npy', which I want to read in batches... – Marlon Teixeira Aug 25 '20 at 14:41
  • is there any hidden thing that we should be aware of when working with memmaps? Is it fine to do operations like if it was a simply numpy array ? – seralouk Dec 09 '21 at 12:15
  • 1
    @seralouk as far as I know it is fine. You should carefully mind about the memory layout and how you work with the `offset` parameter to access the correct position of the arrays. – Saullo G. P. Castro Dec 09 '21 at 12:21
  • @SaulloG.P.Castro thanks. I always use the full matrices (i do not slice it) so in my case it should be fine. – seralouk Dec 09 '21 at 14:23