0

I work with up to ~100000x100000 sparse complex valued matrices (A), with a sparsity of maximum 3%. I wish to do things like expm(-At)v, A\v , etc., where v is an arbitrary vector. I used MATLAB, but I believe this is too much for MATLAB's memory. So I decided to switch to Python HDF5/Pytables.

However, I am unable to find modules in Python Pytables, etc. that do a sparse matrix multiplication in HDF5 style. I don't want to run a for loop over chunks within the sparse matrix, where you borrow a chunk from the matrix, multiply it with an appropriate chunk grabbed from a vector and then move on to the next chunk, without storing the full, humongous matrix in the RAM. The for loop seems too time consuming.

My eventual goal is to integrate this hard-disk based multiplication routine into modules like scipy.sparse.linalg.expm_multiply to calculate expm(-At)*v, etc. for such huge sparse matrices.

user3812405
  • 1
  • I haven't used `pytables` much, but do know that `h5py` does not do any calculations. It just stores/reads data from the file, producing `numpy` arrays. The `scipy.sparse` has a `np.savez` based way of saving sparse matrices (it writes the attributes as arrays). `scipy.io.savemat` can handle sparse matrices, using MATLAB's format. – hpaulj Sep 14 '19 at 19:29
  • Looping through 'chunks' of a sparse matrix is awkward, unless you understand how such a matrix is stored. The `scipy` `csr` 'chunk' indexing is actually performed with matrix multiplication (using an 'extractor' matrix). – hpaulj Sep 14 '19 at 19:53
  • This is similar to another question (and answer). Review this answer for a process to store scipy sparse matrices with HDF5: (https://stackoverflow.com/a/43397032/10462884) – kcw78 Sep 17 '19 at 14:15

0 Answers0