MPI_Recv Hangs While Receiving Large Array

Question

I'm noticing some strange behaviour with a very simple piece of MPI code:

#include <mpi.h>

int main(int argc, char* argv[])
{
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // We are assuming at least 2 processes for this task
    if (world_size != 2)
    {
        std::cout << "World size must be equal to 1" << std::endl;
        MPI_Abort(MPI_COMM_WORLD, 1);
    }

    int numberCounter = 10000;
    double number[numberCounter];

    if (world_rank == 0)
    {
        std::cout << world_rank << std::endl;
        MPI_Send(number, numberCounter, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD);
    }
    else if (world_rank == 1)
    {
        std::cout << world_rank << std::endl;
        MPI_Recv(number, numberCounter, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
    }

    MPI_Finalize();
}

The above works fine provided that numberCounter is small (~1000). When the value is larger however (>10000), the code hangs and never reaches the end. Using MPI_Iprobe does flag that rank 1 has received a message, but MPI_Recv always hangs.

What could be causing this? Can anyone else reproduce this behaviour?

score 0 · Answer 1 · edited May 23 '17 at 12:16

I see 2 immediately concerning problems:

1) You're using a rather large static array.

int numberCounter = 10000;
int number[numberCounter];

Depending on your machine, this may lead to a stack overflow. Segmentation fault on large array sizes

If this is the case, use dynamic arrays. In C, use malloc and free. In C++, I'm not 100% sure new and delete will play nicely with MPI functions, but it should probably work. On C++, you can also consider using the Boost MPI libraries, which work in conjunction with STL containers (like std::vector). http://www.boost.org/doc/libs/1_62_0/doc/html/mpi.html

2) Your send and receive buffers number are int arrays, whereas the MPI functions are instructed to use the MPI_DOUBLE type.

The MPI type should match the array type, so use (double and MPI_DOUBLE) or (int and MPI_INT).

I'm sorry for the sloppy code post - the code I ran does use a double array and not int. Something got missed when copy and pasting.I checked the memory in valgrind and that seems fine also. — Firefly09, Feb 06 '17 at 18:22

MPI_Recv Hangs While Receiving Large Array

1 Answers1