MPI_Send is blocking in ring communication with large data size

Question

I am trying to form a ring communication using MPI, where my each process is sending its result to next process and last process is sending result to 0th process. Lets assume I have 4 processes then my 0th process will send result to 1st, 1st to 2nd, 2nd to 3rd and 3rd to 0th.

#include "mpi.h"
#include <stdio.h>
#include<stdlib.h>
#define NELEM 1000
int main (int argc, char *argv[])
{

    int numtasks, rank, rc, i, dest = 1, tag = 111, source = 0, size;
    double *data, result;
    void *buffer;

    data=(double*)malloc(sizeof(double)*NELEM);
    if(data==NULL)
    {
        printf("Unable to allocate memory\n");
        return;
    }
    MPI_Status status;

    MPI_Init (&argc, &argv);
    MPI_Comm_size (MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank (MPI_COMM_WORLD, &rank);


    for (i = 0; i < NELEM; i++)
        data[i] = (double) random ();

    if (rank == 0)
        source=numtasks-1;
    else
        source=rank-1;
    if(rank==numtasks-1)
        dest=0;
    else
        dest=rank+1;

    printf("Rank %d sending data to rank %d\n",rank,dest);
    MPI_Send(data, NELEM, MPI_DOUBLE, dest, tag,MPI_COMM_WORLD);
    printf("Rank %d send complete\n",rank);

    printf("Rank %d receiving data from rank %d\n",rank,source);
    MPI_Recv (data, NELEM, MPI_DOUBLE, source, tag, MPI_COMM_WORLD,&status);
    printf("Rank %d received data from rank %d\n",rank,source);
    MPI_Finalize ();
}

Here NELEM is the number of elements send or received. If I send less than 100 elements with 4 threads above code will work fine but if I increase the number of thread it will be blocked. I am not getting why it is getting blocked. Is there any restriction on data size which we can send.

Thanks

Ajay

I think you've just run into the dining philosophers problem. — flup, May 02 '13 at 11:55
This is the same question as here: http://stackoverflow.com/questions/15833947/mpi-hangs-on-mpi-send-for-large-messages , with the answer given here http://stackoverflow.com/questions/15833947/mpi-hangs-on-mpi-send-for-large-messages/15837635#15837635 . As @flup points out, the issue is that everyone's sending and no one's receiving, and it's only an accident of common implementations that this works for small messages. As in the answer above, the solution is to use [`MPI_Sendrecv()`](http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Sendrecv.html), or non-blocking communications. — Jonathan Dursi, May 02 '13 at 12:12

flup · Answer 1 · 2013-05-02T14:50:07.590

4

All of your processes are trying to send. But they cannot for none of them is ready to listen.

For smaller element sizes, I expect the message fits inside the buffer.

As Jonathan suggests, the answer is to use MPI_Sendrecv(), or non-blocking communications.

edited May 02 '13 at 14:50

answered May 02 '13 at 11:58

flup

26,937
7
52
74

Yep. Sendrecv. Definitely. – bob.sacamento May 02 '13 at 17:22

MPI_Send is blocking in ring communication with large data size

1 Answers1