1

To my current understanding, after calling MPI_Send, the calling thread should block until the variable is received, so my code below shouldn't work. However, I tried sending several variables in a row and receiving them gradually while doing operations on them and this still worked... See below. Can someone clarify step by step what is going on here?

matlab code: (because I am using a matlab mex wrapper for MPI functions)

%send
if mpirank==0
%arguments to MPI_Send_variable are (variable, destination, tag)
MPI_Send_variable(x,0,'A_22')%thread 0 should block here! 
MPI_Send_variable(y,0,'A_12')
MPI_Send_variable(z,1,'A_11')
MPI_Send_variable(w,1,'A_21')
end

%recieve
if mpirank==0
%arguments to MPI_Recv_variable are (source, tag)
a=MPI_Recv_variable(0,'A_12')*MPI_Recv_variable(0,'A_22');
end
if mpirank==1
c=MPI_Recv_variable(0,'A_21')*MPI_Recv_variable(0,'A_22');
end
Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
avgn
  • 982
  • 6
  • 19
  • How do a process can ever receive a message that he sent to himself if both the send and receive operations are blocking ? Either send to self and receive to self are non-blocking or you will get a deadlock. According to http://stackoverflow.com/questions/11385395/is-the-behavior-of-mpi-communication-of-a-rank-with-itself-well-defined , it seems that the first alternative is the standard one. – francis Aug 25 '14 at 18:10

3 Answers3

2

MPI_SEND is a blocking call only in the sense that it blocks until it is safe for the user to use the buffer provided to it. The important text to read here is in Section 3.4:

The send call described in Section 3.2.1 uses the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the data has been moved to the receiver.

I highlighted the part that you're running up against in bold there. If your message is sufficiently small (and there are sufficiently few of them), MPI will copy your send buffers to an internal buffer and keep track of things internally until the message has been received remotely. There's no guarantee that when MPI_SEND is done, the message has been received.

On the other hand, if you do want to know that the message was actually received, you can use MPI_SSEND. That function will synchronize (hence the extra S both sides before allowing them to return from the MPI_SSEND and the matching receive call on the other end.

Wesley Bland
  • 8,816
  • 3
  • 44
  • 59
  • To be clear, MPI_SSEND only guarantees that the message has been matched on the receiver side. It does not guarantee that any bytes have been moved yet. – Pavan Balaji Aug 11 '18 at 10:10
1

I assume that there is just a MPI_Send() behind MPI_Send_variable() and MPI_Receive() behind MPI_Receive_variable().

How do a process can ever receive a message that he sent to himself if both the send and receive operations are blocking ? Either send to self or receive to self are non-blocking or you will get a deadlock, and sending to self is forbidden.

Following answer of @Greginozemtsev Is the behavior of MPI communication of a rank with itself well-defined? , the MPI standard states that send to self and receive to self are allowed. I guess it implies that it's non blocking in this particular case.

In MPI 3.0, in section 3.2.4 Blocking Receive here, page 59, the words have not changed since MPI 1.1 :

 Source = destination is allowed, that is, a process can send a message to itself.
     (However, it is unsafe to do so with the blocking send 
     and receive operations described above, since this may lead to deadlock. 
     See Section 3.5.)

I rode section 3.5, but it's not clear enough for me...

I guess that the parenthesis are here to tell us that talking to oneself is not a good practice, at least for MPI communications !

Community
  • 1
  • 1
francis
  • 9,525
  • 2
  • 25
  • 41
1

In a correct MPI program, you cannot do a blocking send to yourself without first posting a nonblocking receive. So a correct version of your program would look something like this:

Irecv(..., &req1);
Irecv(..., &req2);

Send(... to self ...);
Send(.... to self ...);

Wait(&req1, ...);
/* do work */
Wait(&req2, ...);
/* do more work */

Your code is technically incorrect, but the reason it is working correctly is because the MPI implementation is using internal buffers to buffer your send data before it is transmitted to the receiver (or matched to the later receive operation in the case of self sends). An MPI implementation is not required to have such buffers (generally called "eager buffers"), but most implementations do.

Since the data you are sending is small, the eager buffers are generally sufficient to buffer them temporarily. If you send large enough data, the MPI implementation will not have enough eager buffer space and your program will deadlock. Try sending, for example, 10 MB instead of a double in your program to notice the deadlock.

Pavan Balaji
  • 81
  • 1