MPI all to all operation with one part of a huge matrix at a time

Question

I have a distributed matrix over 4 nodes and I want every node to send its part of the matrix and also receive, one at a time, every other part of the matrix from other nodes. The block matrices have different dimensions.

I tried to write some code, but it's not working as expected.

  /* send my part of the matrix */
  for (int i = 0; i < numtasks; i++){
    if (i == taskid) continue;

    MPI_Isend(matrix_block, size, MPI_INT, i, 0,
              MPI_COMM_WORLD, &rNull);
  }

  /* receive everyone's part of the matrix */
  for (int i = 0; i < numtasks; i++){
    if (i == taskid) continue;

    MPI_Irecv(brec, lenghts_recv[i], MPI_INT, i, 0,
              MPI_COMM_WORLD, &request[i]);
  }

  for (int i = 0; i < numtasks - 1; i++){
    int index;
    MPI_Waitany(numtasks-1, request, &index, &status);
  }

I supposed every node will first send the block it has and after that it will receive what has been send to him by other nodes, but obviously something it's wrong.

Also, solutions like MPI_Alltoall do not work in my case because the matrix it's supposed to be huge and it won't fit inside one node.

Can you suggest me a way of performing the all to all operation but with one part of the matrix at a time?

score 1 · Answer 1 · edited May 23 '17 at 11:53

You can use MPI_Bcast to let each of the four nodes send its part of the matrix to the other three. This way, you can split the all-to-all operation into several one-to-all operations that you can interleave with computation.

So basically, you could do:

 for (int i = 0; i < numtasks; i++){
     /* process i sends the data in matrix_block to all other processes. This is a 
        collective operation, i.e., after the operation, every process will have 
        already received the data into matrix_block. */
     MPI_Bcast(matrix_block, size, MPI_INT, i, MPI_COMM_WORLD);

     //TODO: do all necessary computation on this part of the matrix */
}

I'm not sure how your code works and what all the variables are, so I can't give you anything more concrete. If you update your questions with a minimum working example, I might be able to help more.

You can find an example using MPI_Bcast in this excellent answer.

MPI all to all operation with one part of a huge matrix at a time

1 Answers1