The most efficient way to store large symmetric sparse matrices in python

Question

I was working on drafting/testing a technique I devised for solving differential equations for speed and efficiency.

It would require a storing, manipulating, resizing, and (at some point) probably diagonalizing very large sparse matrices. I would like to be able to have rows consisting of zeros and a few (say <5) ones, and add them a few at a time (on the order of the number of cpus being used.)

I thought it would be useful to have gpu accelleration--so any suggestions as to the best way to take advange of that would be appreciated too (say pycuda, theano, etc.)

or try [scipy.sparse](http://docs.scipy.org/doc/scipy/reference/sparse.html)? — nye17, Aug 29 '11 at 02:40
I was anticipating something like this: http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices — Alex Eftimiades, Aug 29 '11 at 02:52
I have used numpy and a bit of scipy. I mainly wanted to check that these were indeed the most scalable options--and also for any tips on using the gpu. — Alex Eftimiades, Aug 29 '11 at 02:54

score 7 · Answer 1 · answered Aug 29 '11 at 03:46

The most efficient storage method for symmetric sparse matrices is probably sparse skyline format (this is what Intel MKL uses, for example). AFAIK scipy.sparse doesn't contain a sparse, symmetric matrix format. Pysparse, however, does. Using Pysparse, you can build the matrix incrementally using the link list format, then convert the matrix into sparse skyline format. Performance wise, I have usually found Pysparse to be superior to scipy with large sparse systems, and all the basic building blocks (matrix product, eigenvalue solver, direct solver, iterative solver) as present, although the range of routines is perhaps a little smaller than what is available in scipy.

JBernardo · Accepted Answer · 2013-07-05T02:29:18.607

4

You can use a dictionary and tuples to access the data:

>>> size = (4,4)
>>> mat = {}
>>> mat[0,1] = 3
>>> mat[2,3] = 5
>>> for i in range(size[0]):
        for j in range(size[1]):
            print mat.get((i,j), 0) ,
        print

0 3 0 0
0 0 0 0
0 0 0 5
0 0 0 0

Of course you should make a class for that and add the methods you need:

class Sparse(dict):
    pass

BTW, You can also use scipy.sparse from the scipy lib

edited Jul 05 '13 at 02:29

answered Aug 29 '11 at 02:42

JBernardo

32,262
10
90
115

Ok, if that would be most efficient then so be it. Thank you, I mainly wanted to check for obscure answers. – Alex Eftimiades Aug 29 '11 at 02:52
I will just give the question another hour or so to see if any other answers pop up then I will mark this as solved. Thanks. – Alex Eftimiades Aug 29 '11 at 02:54
@Feynman A Sparse matrix class should overhide the `__getitem__` method to return zero when there's no data – JBernardo Aug 29 '11 at 02:56
thanks for the tip. I am still pretty new to python. I will be sure to override the getitem method. I guess I will base the Sparse class on the scipy sparse class and modify some of the built in stuff. – Alex Eftimiades Aug 29 '11 at 03:49

score 3 · Answer 3 · answered Aug 29 '11 at 02:41

3

Use scipy.sparse.

answered Aug 29 '11 at 02:41

John Zwinck

239,568
38
324
436

The most efficient way to store large symmetric sparse matrices in python

3 Answers3