4

I am trying to calculate the dot product between two large Scipy Sparse matrices (5million x 5million product with 5million x 10000). Both matrices fit in memory and I have a 40 core system. However, the dot product operation of Scipy does not parallelise across any more than 1 core.

Currently it is taking a lot of time to run this simple computation. I have read the Scipy documentation, along with this page: https://scipy.github.io/old-wiki/pages/ParallelProgramming , however, it is nowhere mentioned that the dot operation has a parallel implementation.

I also tried installing OpenMP, BLAS, LAPACK etc. installations of the libraries but none of them seems to provide any parallel processing capabilities for Scipy Sparse matrices. However, I can confirm that Numpy dot product is being parallelised well.

  1. Can it be confirmed that Sparse matrix product in Scipy is not parallelised?
  2. Is there any possible way to parallelise this operation to get the product faster than running it on one core?
hpaulj
  • 221,503
  • 14
  • 230
  • 353
Shivam Khandelwal
  • 121
  • 4
  • 1
    See https://stackoverflow.com/q/37536106/901925 – hpaulj Aug 19 '20 at 15:08
  • I would second using mkl but mkl_sparse_spmm would be the function to call in mkl 2020 – CJR Aug 20 '20 at 03:28
  • 1
    Also there's at least one python library that wraps mkl for some sparse functions https://pypi.org/project/sparse-dot-mkl/ – CJR Aug 20 '20 at 03:30

0 Answers0