Rewrite several matrix operations to avoid memory error?

Question

I have a very big matrix. buildmatrix is an N by N matrix. So sumHamiltonian is a N^2 by N^2 matrix. Then transform(N) is an N^2 by N(N-1)/2 matrix. (So the resultant shortHamiltonian(N) is an N(N-1)/2 by N(N-1)/2 matrix). The matrix entries are also complex numbers.

If N=200, I get a memory error. Is there a way to rewrite:

def sumHamiltonian(N):
    return 0.5*(np.kron(buildmatrix(N),np.identity(N))+np.kron(np.identity(N),buildmatrix(N)))

def shortHamiltonian(N):
    return np.matmul(np.transpose(transform(N)),np.matmul(sumHamiltonian(N),transform(N)))

to reduce memory?

I have seen some ways to reduce memory for matrix multiplication (Python/Numpy MemoryError) which is helpful but my memory error comes up at the kronecker product. Is there a way to rewrite this, or better yet, all of the matrix operations to avoid a memory error?

With a small N I make the arrays, and look for ways I break them up into chunks. Also the `sum` is a mean of 2 similar `kron`.. Those `kron` may be sparce due to `np.idenity`. I'd look for patterns specific to this case, rather hope for some generic 'memory saver'. N^4 gets big fast, especially if you are summing two. — hpaulj, Feb 24 '20 at 05:39

Andreas K. · Accepted Answer · 2020-02-24T08:27:51.280

2

One suggestion is to use sparse matrices. Also, if possible, make your buildmatrix to return a matrix of dtype np.complex64 instead of np.complex128. See the example below:

from scipy import sparse


def buildmatrix(N):
    return (np.random.rand(N, N) + np.random.rand(N, N) * 1j).astype(np.complex64)

N = 100

m = buildmatrix(N)
I = np.identity(N)
sumHamiltonian = 0.5 * (np.kron(m, I) + np.kron(I, m))
print(f'{sumHamiltonian.nbytes * 1e-9} GB')
#1.6 GB

m_s = sparse.csr_matrix(m)
I_s = sparse.identity(N)
sumHamiltonian_s = 0.5 * (sparse.kron(m_s, I_s) + sparse.kron(I_s, m_s))
print(f'{sumHamiltonian_s.data.nbytes * 1e-6} MB')
#31.84 MB

#np.all(sumHamiltonian == sumHamiltonian_s)
#True

Similarly try to convert your shortHamiltonian to "sparse version".

edited Feb 24 '20 at 08:27

answered Feb 24 '20 at 08:22

Andreas K.

9,282
3
40
45

Sparse matrices has changed my life - thanks so much for the suggestion, this was exactly what I was after! – Janet Zhong Mar 02 '20 at 14:07
Glad it helped! Please, consider accepting my answer. – Andreas K. Mar 02 '20 at 16:24

score 1 · Answer 2 · answered Feb 24 '20 at 05:51

Depending on how much memory excess you have, you may be able to decrease the size of your matrices just by using something like .astype(np.float32) (e.g. if the current dtype is np.float64).

buildmatrix(N) is called twice in the same line (same with transform). If the result is deterministic (you just want to use the same result) then try assigning it to a variable, so that it only needs to be called once.

These tips might slightly reduce memory profile but might or might not be enough. I'm not sure if the computation can also be simplified by some identity.

Rewrite several matrix operations to avoid memory error?

2 Answers2