MPI derived datatype problem caused by struct padding and non-blocking communication buffer's problem

Question

Hi I am writing a c++ program, in which I want MPI to communicate by a derived data type. But the receiver does not receive the full information that the sender sends out.

Here is how I build my derived data type:

// dg_derived_datatype.cpp

#include <mpi.h>
#include "dg_derived_datatype.h"

namespace Hash{

    MPI_Datatype Face_type;
};

void Construct_data_type(){

    MPI_Face_type();

}

void MPI_Face_type(){

    int num = 3;

    // Number of elements in each block (array of integers)
    int elem_blocklength[num]{2, 1, 5};

    // Byte displacement of each block (array of integers).
    MPI_Aint array_of_offsets[num];
    MPI_Aint intex, charex;
    MPI_Aint lb;
    MPI_Type_get_extent(MPI_INT, &lb, &intex);
    MPI_Type_get_extent(MPI_CHAR, &lb, &charex);

    array_of_offsets[0] = (MPI_Aint) 0;
    array_of_offsets[1] = array_of_offsets[0] + intex * 2;
    array_of_offsets[2] = array_of_offsets[1] + charex;

    MPI_Datatype array_of_types[num]{MPI_INT, MPI_CHAR, MPI_INT};

    // create and MPI datatype
    MPI_Type_create_struct(num, elem_blocklength, array_of_offsets, array_of_types, &Hash::Face_type);  
    MPI_Type_commit(&Hash::Face_type);

}

void Free_type(){

    MPI_Type_free(&Hash::Face_type);    

}

Here I derive my data type Hash::Face_type and commit it. The Hash::Face_type is used to transfer my struct (face_pack, 2 int + 1 char + 5 int) vector.

// dg_derived_datatype.h

#ifndef DG_DERIVED_DATA_TYPE_H
#define DG_DERIVED_DATA_TYPE_H

#include <mpi.h>

struct face_pack{

    int owners_key; 

    int facei; 

    char face_type;

    int hlevel;

    int porderx;

    int pordery; 

    int key;

    int rank;

};

namespace Hash{

    extern MPI_Datatype Face_type;
};

void Construct_data_type();

void Free_type();

#endif

Then in my main program I do

// dg_main.cpp

#include <iostream>
#include <mpi.h>
#include "dg_derived_datatype.h"
#include <vector>

void Recv_face(int source, int tag, std::vector<face_pack>& recv_face);

int main(){
// Initialize MPI. 
// some code here.
// I create a vector of struct: std::vector<face_pack> face_info,
// to store the info I want to let proccesors communicate. 

Construct_data_type(); // construct my derived data type

MPI_Request request_pre1, request_pre2, request_next1, request_next2;

// send
if(num_next > 0){ // If fullfilled the current processor send info to the next processor (myrank + 1)

std::vector<face_pack> face_info;
// some code to construct face_info

// source my_rank, destination my_rank + 1
MPI_Isend(&face_info[0], num_n, Hash::Face_type, mpi::rank + 1, mpi::rank + 1, MPI_COMM_WORLD, &request_next2);

}

// recv
if(some critira){ // recv from the former processor (my_rank - 1)

std::vector<face_pack> recv_face;

Recv_face(mpi::rank - 1, mpi::rank, recv_face); // recv info from former processor

}
if(num_next > 0){

        MPI_Status status;
        MPI_Wait(&request_next2, &status);

}

Free_type();

// finialize MPI
}

void Recv_face(int source, int tag, std::vector<face_pack>& recv_face){

    MPI_Status status1, status2;

    MPI_Probe(source, tag, MPI_COMM_WORLD, &status1);

    int count;
    MPI_Get_count(&status1, Hash::Face_type, &count);

    recv_face = std::vector<face_pack>(count);

    MPI_Recv(&recv_face[0], count, Hash::Face_type, source, tag, MPI_COMM_WORLD, &status2);
}

The problem is that the receiver sometimes receives an incomplete info.

For example, I print out the face_info before it is sent out:

// rank 2
owners_key3658 facei 0 face_type M neighbour 192 n_rank 0
owners_key3658 facei 1 face_type L neighbour 66070 n_rank 1
owners_key3658 facei 1 face_type L neighbour 76640 n_rank 1
owners_key3658 facei 2 face_type M neighbour 2631 n_rank 0
owners_key3658 facei 3 face_type L neighbour 4953 n_rank 1
...
owners_key49144 facei 1 face_type M neighbour 844354 n_rank 2
owners_key49144 facei 1 face_type M neighbour 913280 n_rank 2
owners_key49144 facei 2 face_type L neighbour 41619 n_rank 1
owners_key49144 facei 3 face_type M neighbour 57633 n_rank 2

Which is correct.

But in the receiver side, I print out the message it received:

owners_key3658 facei 0 face_type M neighbour 192 n_rank 0
owners_key3658 facei 1 face_type L neighbour 66070 n_rank 1
owners_key3658 facei 1 face_type L neighbour 76640 n_rank 1
owners_key3658 facei 2 face_type M neighbour 2631 n_rank 0
owners_key3658 facei 3 face_type L neighbour 4953 n_rank 1

... // at the beginning it's fine, however, at the end it messed up

owners_key242560 facei 2 face_type ! neighbour 2 n_rank 2
owners_key217474 facei 2 face_type ! neighbour 2 n_rank 2
owners_key17394 facei 2 face_type ! neighbour 2 n_rank 2
owners_key216815 facei 2 face_type ! neighbour 2 n_rank 2

Surely, it lost the face_type info, which is a char. And as far as I know, the std::vector warrants a contiguous memory here. So I am not sure which part of my derived mpi datatype is wrong. The message passing sometimes works sometimes not.

you'd rather use the `offsetof()` macro. Note you might also have to `MPI_Type_create_resized()` your datatype to take into account the last padding (e.g. `gap_1`). Creating the MPI derived datatype once for all should be enough (assuming all your files were compiled with the same options, so they all use the same padding). — Gilles Gouaillardet, Mar 23 '20 at 00:50
Thank you, Gilles. I now use `MPI_Type_create_resized`. But I still have the same problem. I found out that the program crashes if the `recv_face` vector becomes large, say its size reaches 150. I still not sure why. — Shiqi, Mar 23 '20 at 01:49

score 1 · Accepted Answer · answered Mar 24 '20 at 00:37

OK, I kind of figured out my problems. There are two.

The first one is the use of MPI_Type_get_extent(). Since c/c++ struct could be padded by your compiler, it is OK that if you only send one element, but if you send multiple elements, the trailing padding could cause problems (see picture below).

Therefore, a safer and a more protable way to define your derived datatype is to use MPI_Get_address(). Here is how I do it:

// generate the derived datatype
void MPI_Face_type(){

    int num = 3;

    int elem_blocklength[num]{2, 1, 5};

    MPI_Datatype array_of_types[num]{MPI_INT, MPI_CHAR, MPI_INT};

    MPI_Aint array_of_offsets[num];
    MPI_Aint baseadd, add1, add2;

    std::vector<face_pack> myface(1);

    MPI_Get_address(&(myface[0].owners_key), &baseadd);
    MPI_Get_address(&(myface[0].face_type), &add1);
    MPI_Get_address(&(myface[0].hlevel), &add2);

    array_of_offsets[0] = 0;
    array_of_offsets[1] = add1 - baseadd;
    array_of_offsets[2] = add2 - baseadd;

    MPI_Type_create_struct(num, elem_blocklength, array_of_offsets, array_of_types, &Hash::Face_type);  

    // check that the extent is correct
    MPI_Aint lb, extent;
    MPI_Type_get_extent(Hash::Face_type, &lb, &extent); 
    if(extent != sizeof(myface[0])){
        MPI_Datatype old = Hash::Face_type;
        MPI_Type_create_resized(old, 0, sizeof(myface[0]), &Hash::Face_type);
        MPI_Type_free(&old);
    }
    MPI_Type_commit(&Hash::Face_type);
}

The second one is the use of non-blocking send MPI_Isend(). The program works properly after I changed the non-blocking send to blocking send.

The relative part of my program looks like this:

if(criteria1){

//form the vector using my derived datatype
std::vector<derived_type> my_vector;

// use MPI_Isend to send the vector to the target rank
MPI_Isend(... my_vector...);

}

if(critira2){

// need to recv message 
MPI_Recv();
}

if(critira1){

// the sender now needs to make sure the message has arrived. 
MPI_Wait();
}

Although I used MPI_Wait the recver did not get the full message. I check the man page of MPI_Isend(), it says man_page

A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not modify any part of the send buffer after a nonblocking send operation is called until the send completes.

But I do not think I modified the send buffer? Or it could be there is no enough space in the send buffer to store the info to be sent? In my understanding, the non-blocking send works like this, the sender put the message in its buffer and send out to the target rank when the target rank hits MPI_Recv. So it could be the sender's buffer runs out of space to store messages before it sends them out? Correct me if I am wrong.

note you can use the `offsetof()` macro to fill `array_of_offsets`, or you might also use `MPI_Aint_diff()` — Gilles Gouaillardet, Mar 24 '20 at 01:00
If you `MPI_Isend()`it is up to you **not** to modify the send buffer until the message is sent (e.g. After `MPI_Wait()` completed or `MPI_Test()` says so). Never assume the message will be sent in eager mode and you can skip waiting for completion) — Gilles Gouaillardet, Mar 24 '20 at 11:01

MPI derived datatype problem caused by struct padding and non-blocking communication buffer's problem

1 Answers1