Questions tagged [matrix-multiplication]

Questions related to matrix multiplication, especially implementation. Mathematical questions should consider the [linear-algebra] tag.

Matrix multiplication is usually the process of multiplying two (or more) matrices. This arises quite frequently in linear algebra contexts and is a particularly fundamental task in computing, especially in scientific computing.

To that end, a number of fundamental libraries, such as LAPACK, BLAS, ATLAS, and others have been developed. Because the growth of matrices affects the computational time, extensive effort has been made to optimize these packages for various computer architectures and various matrix sizes.

In scientific software for statistical computing and graphics, operator %*% performs matrix multiplication (see ?"%*%"), and interfaces BLAS routine dgemmm.


The product of the multiplication of two matrices a and b is the matrix c, where each element is the sum of the products of the i-th row of a and the j-th column of b.

c[i][j] += a[i][k] * b[k][j];

Example:

     (b) { 1,  2,  3,  4}
         { 5,  6,  7,  8}
(a)    ┌(c)──────────────
{1, 2} │ {11, 14, 17, 20}
{3, 4} │ {23, 30, 37, 44}
{5, 6} │ {35, 46, 57, 68}

Algorithm of the matrix multiplication:

// rows of 'a' matrix
int m = 3;
// columns of 'a' matrix
// and rows of 'b' matrix
int n = 2;
// columns of 'b' matrix
int p = 4;
// matrices 'a=m×n', 'b=n×p'
int[][] a = {{1, 2}, {3, 4}, {5, 6}},
        b = {{1, 2, 3, 4}, {5, 6, 7, 8}};
// resulting matrix 'c=m×p'
int[][] c = new int[m][p];
// iterate over the rows of the 'a' matrix
for (int i = 0; i < m; i++) {
    // iterate over the columns of the 'b' matrix
    for (int j = 0; j < p; j++) {
        // iterate over the columns of the 'a'
        // matrix, aka rows of the 'b' matrix
        for (int k = 0; k < n; k++) {
            // sum of the products of
            // the i-th row of 'a' and
            // the j-th column of 'b'
            c[i][j] += a[i][k] * b[k][j];
        }
    }
}
2901 questions
232
votes
4 answers

What is the '@=' symbol for in Python?

I know @ is for decorators, but what is @= for in Python? Is it just reservation for some future idea? This is just one of my many questions while reading tokenizer.py.
205
votes
12 answers

Why is MATLAB so fast in matrix multiplication?

I am making some benchmarks with CUDA, C++, C#, Java, and using MATLAB for verification and matrix generation. When I perform matrix multiplication with MATLAB, 2048x2048 and even bigger matrices are almost instantly multiplied. …
Wolf
  • 3,117
  • 3
  • 19
  • 10
188
votes
6 answers

Difference between numpy dot() and Python 3.5+ matrix multiplication @

I recently moved to Python 3.5 and noticed the new matrix multiplication operator (@) sometimes behaves differently from the numpy dot operator. In example, for 3d arrays: import numpy as np a = np.random.rand(8,13,13) b = np.random.rand(8,13,13) c…
blaz
  • 4,108
  • 7
  • 29
  • 54
173
votes
4 answers

How to get element-wise matrix multiplication (Hadamard product) in numpy?

I have two matrices a = np.matrix([[1,2], [3,4]]) b = np.matrix([[5,6], [7,8]]) and I want to get the element-wise product, [[1*5,2*6], [3*7,4*8]], equaling [[5,12], [21,32]] I have tried print(np.dot(a,b)) and print(a*b) but both give the…
Malintha
  • 4,512
  • 9
  • 48
  • 82
146
votes
8 answers

How does BLAS get such extreme performance?

Out of curiosity I decided to benchmark my own matrix multiplication function versus the BLAS implementation... I was to say the least surprised at the result: Custom Implementation, 10 trials of 1000x1000 matrix multiplication: Took: 15.76542…
DeusAduro
  • 5,971
  • 5
  • 29
  • 36
144
votes
8 answers

how does multiplication differ for NumPy Matrix vs Array classes?

The numpy docs recommend using array instead of matrix for working with matrices. However, unlike octave (which I was using till recently), * doesn't perform matrix multiplication, you need to use the function matrixmultipy(). I feel this makes the…
elexhobby
  • 2,588
  • 5
  • 24
  • 33
130
votes
10 answers

Why is there huge performance hit in 2048x2048 versus 2047x2047 array multiplication?

I am making some matrix multiplication benchmarking, as previously mentioned in Why is MATLAB so fast in matrix multiplication? Now I've got another issue, when multiplying two 2048x2048 matrices, there is a big difference between C# and others.…
Wolf
  • 3,117
  • 3
  • 19
  • 10
88
votes
4 answers

How do I multiply matrices in PyTorch?

With numpy, I can do a simple matrix multiplication like this: a = numpy.ones((3, 2)) b = numpy.ones((2, 1)) result = a.dot(b) However, this does not work with PyTorch: a = torch.ones((3, 2)) b = torch.ones((2, 1)) result = torch.dot(a, b) This…
timbmg
  • 3,192
  • 7
  • 34
  • 52
82
votes
5 answers

Matrix multiplication: Small difference in matrix size, large difference in timings

I have a matrix multiply code that looks like this: for(i = 0; i < dimension; i++) for(j = 0; j < dimension; j++) for(k = 0; k < dimension; k++) C[dimension*i+j] += A[dimension*i+k] * B[dimension*k+j]; Here, the size of the …
Korizon
  • 3,677
  • 7
  • 37
  • 52
70
votes
5 answers

CUDA determining threads per block, blocks per grid

I'm new to the CUDA paradigm. My question is in determining the number of threads per block, and blocks per grid. Does a bit of art and trial play into this? What I've found is that many examples have seemingly arbitrary number chosen for these…
dnbwise
  • 1,092
  • 1
  • 9
  • 20
69
votes
5 answers

Vectorized way of calculating row-wise dot product two matrices with Scipy

I want to calculate the row-wise dot product of two matrices of the same dimension as fast as possible. This is the way I am doing it: import numpy as np a = np.array([[1,2,3], [3,4,5]]) b = np.array([[1,2,3], [1,2,3]]) result = np.array([]) for…
Cupitor
  • 11,007
  • 19
  • 65
  • 91
66
votes
4 answers

2-D convolution as a matrix-matrix multiplication

I know that, in the 1D case, the convolution between two vectors, a and b, can be computed as conv(a, b), but also as the product between the T_a and b, where T_a is the corresponding Toeplitz matrix for a. Is it possible to extend this idea to…
64
votes
4 answers

Faster way to initialize arrays via empty matrix multiplication? (Matlab)

I've stumbled upon the weird way (in my view) that Matlab is dealing with empty matrices. For example, if two empty matrices are multiplied the result is: zeros(3,0)*zeros(0,3) ans = 0 0 0 0 0 0 0 0 0 Now, this already…
bla
  • 25,846
  • 10
  • 70
  • 101
64
votes
6 answers

Why is matrix multiplication faster with numpy than with ctypes in Python?

I was trying to figure out the fastest way to do matrix multiplication and tried 3 different ways: Pure python implementation: no surprises here. Numpy implementation using numpy.dot(a, b) Interfacing with C using ctypes module in Python. This is…
Charles Menguy
  • 40,830
  • 17
  • 95
  • 117
57
votes
1 answer

What is R's multidimensional equivalent of rbind and cbind?

When working with matrices in R, one can put them side-by-side or stack them top of each other using cbind and rbind, respectively. What is the equivalent function for stacking matrices or arrays in other dimensions? For example, the following…
Ryan C. Thompson
  • 40,856
  • 28
  • 97
  • 159
1
2 3
99 100