33

I wonder when do I need to use barrier? Do I need it before/after a scatter/gather for example? Or should OMPI ensure all processes have reached that point before scatter/gather-ing? Similarly, after a broadcast can I expect all processes to already receive the message?

Paul Price
  • 2,657
  • 30
  • 26
Jiew Meng
  • 84,767
  • 185
  • 495
  • 805

3 Answers3

31

All collective operations in MPI before MPI-3.0 are blocking, which means that it is safe to use all buffers passed to them after they return. In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!) So MPI_Barrier is not necessary (or very helpful) before/after collective operations, if all buffers are valid already.

Please also note, that MPI_Barrier does not magically wait for non-blocking calls. If you use a non-blocking send/recv and both processes wait at an MPI_Barrier after the send/recv pair, it is not guaranteed that the processes sent/received all data after the MPI_Barrier. Use MPI_Wait (and friends) instead. So the following piece of code contains errors:

/* ERRORNOUS CODE */

Code for Process 0:
Process 0 sends something using MPI_Isend
MPI_Barrier(MPI_COMM_WORLD);
Process 0 uses buffer passed to MPI_Isend // (!)

Code for Process 1:
Process 1 recvs something using MPI_Irecv
MPI_Barrier(MPI_COMM_WORLD);
Process 1 uses buffer passed to MPI_Irecv // (!)

Both lines that are marked with (!) are unsafe!

MPI_Barrier is only useful in a handful of cases. Most of the time you do not care whether your processes sync up. Better read about blocking and non-blocking calls!

Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
Markus Mayr
  • 4,038
  • 1
  • 20
  • 42
  • Why is the 1st `(!)` an error? Process 0 will still have its own buffer? Also since its a send, the receiving party will not change it right? – Jiew Meng Nov 09 '12 at 10:34
  • 2
    @JiewMeng MPI must not read from the buffer immediately after you call MPI_Isend. If you change it at `(!)`, you might send something different. I am not quite sure about it, but I think that behaviour is undefined in this case. – Markus Mayr Nov 09 '12 at 10:36
  • 3
    I've updated slightly your answer as MPI-3.0 introduced non-blocking collectives. – Hristo Iliev Nov 09 '12 at 11:53
  • "In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!)" - isn't it inconsistent? How can all data be received without being sent? Maybe you've meant that because all collective operations are blocking, it's safe to reuse a buffer with the data-to-sent after a `send` call (because that's what "blocking" is about), because it's "copied" by MPI (not necessarily in the same way as for buffered send `MPI_Bsend`)? Of course it's correct that when blocking send returns we can't be sure that the data was received. – falconepl Mar 13 '15 at 14:59
  • And to make it clear... although with blocking send we can't be sure that the data was received, if we've used **synchronized** blocking send then when send returns we are certain that the recipient has received our message. – falconepl Mar 13 '15 at 15:05
  • 1
    @falconepl You are right, that's what I am saying. In my opinion the wording is not inconsistent, but I hope your comments improve clarity for people who feel the same way like you. Thank you! Just to repeat this once more: A blocking send does not imply that the message was sent and received, just that you can reuse buffers. A blocking receive call implies that all data was received. – Markus Mayr Jun 02 '15 at 05:45
  • @MarkusMayr Hi, MarkusMayr, Your saying still confused me after I read it several times... What does "However, it does not imply that all data was sent!" mean?? Could you please explain it more clearly? – user15964 Mar 25 '16 at 00:54
  • @user15964: Your MPI implementation may create a copy of the buffers you passed to the blocking call. It may use that copy to send the data at a later point in time. For a collective operation, it is possible that your process has not finished sending data to other processes, but it already received all data that it needs, when the blocking call returns. – Markus Mayr Mar 25 '16 at 08:47
  • @MarkusMayr That is much clearer, I think I understand your point now. Thank you very much! – user15964 Mar 26 '16 at 12:07
  • Thank you for the answer. Could you throw some light on whether MPI_Barrier is needed after calling topology functions such as MPI_Cart_Create (and before further processing of it with MPI_Cart_Shift or with MPI_Scatterv/Gatherv for example)? – Ombrophile Dec 03 '19 at 12:31
23

One use of MPI_Barrier is for example to control access to an external resource such as the filesystem, which is not accessed using MPI. For example, if you want each process to write stuff to a file in sequence, you could do it like this:

int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
for ( int ii = 0; ii < size; ++ii ) {
    if ( rank == ii ) {
        // my turn to write to the file
        writeStuffToTheFile();
    }
    MPI_Barrier(MPI_COMM_WORLD);
}

That way, you can be sure that no two processes are concurrently calling writeStuffToTheFile.

Edric
  • 23,676
  • 2
  • 38
  • 40
  • this is great for illustrative purpose. For this particular use case though, I'm wondering if there's some better, more efficient way? – Shihab Shahriar Khan Mar 13 '23 at 23:45
  • 1
    Yes, there is a whole range of MPI-IO support available for parallel file access if you are doing something more substantial. – Edric Mar 15 '23 at 06:49
0

May MPI_Barrier() is not often used, but it is useful. In fact, even if you were use the synchronous communication, the MPI_Send/Recv() can only make sure the two processes is synchronized. In my project, a cuda+MPI project, all i used is asynchronous communication. I found that in some cases if i dont use the MPI_Barrier() followed by the Wait() function, the situation that two processes(gpu) want to transmit data to each other at the same time is very likely to happen, which could badly reduce the program efficiency. The bug above ever divers me mad and take me a few days to find it. Therefore you may think carefully whether use the MPI_Barrier() when you used the MPI_Isend/Irecv in your program. Sometimes sync the processes is not only neccessary but also MUST, especially ur program is dealing with the device.

Freezenfire
  • 17
  • 1