MPI_File_write_at : Writing the same struct twice results in slightly different data blocks in binary file

Question

I have the following simple MPI code:

#include <iostream>
#include <mpi.h>

int main() {
    struct test {
        int rank = 1;
        double begin = 0.5;
        double end = 0.5;
    };

    MPI_Init(NULL, NULL);
    
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    MPI_File fh;
    MPI_Offset offset;
    MPI_Status status;
    MPI_Datatype TIMEDATA_DATATYPE;

    // Create datatype
    int blocklen[3] = {1,1,1};
    MPI_Datatype type[3] = { MPI_INT, MPI_DOUBLE, MPI_DOUBLE };
    MPI_Aint disp[3] = {0, 4, 12};
    MPI_Type_create_struct(3, blocklen, disp, type, &TIMEDATA_DATATYPE);
    MPI_Type_commit(&TIMEDATA_DATATYPE);

    // Open file
    offset = 20*world_rank;

    MPI_File_open(MPI_COMM_WORLD, "test.bin", MPI_MODE_CREATE | MPI_MODE_WRONLY,
            MPI_INFO_NULL, &fh);

    // Write to file
    test t1, t2;
    MPI_File_write_at(fh, offset, &t1, 1, TIMEDATA_DATATYPE, &status);
    MPI_File_write_at(fh, offset, &t2, 1, TIMEDATA_DATATYPE, &status);

    // Close file
    MPI_File_close(&fh);

    MPI_Finalize();
    
    return 0;
}

Run and compiled with

mpic++ Test.cpp
mpirun -n 2 a.out

So in the code above, I basically want to write to the same file twice. Each time from a different process. The data is a struct of the format int, double, double so 4+8+8=20 bytes of data. We have two objects of that struct but both are initialized with the same values. So we write two blocks of 20 bytes of data into the file test.bin and I expect to see this "symmetry" in the binary representation but I get: (I provide two different outputs of two different cli tools.)

$ xxd test.bin 
00000000: 0100 0000 fe7f 0000 0000 0000 0000 e03f  ...............?
00000010: 0000 e03f 0100 0000 ff7f 0000 0000 0000  ................
00000020: 0000 e03f 0000 0000                      ...?....

$ hexdump test.bin 
0000000 0001 0000 7ffe 0000 0000 0000 0000 3fe0
0000010 0000 0000 0001 0000 7fff 0000 0000 0000
0000020 0000 3fe0 0000 0000                    
0000028

Now, if we look at the output of xxd we see:

First 20 bytes:

Integer: 0100 0000
Double: fe7f 0000 0000 0000
Double: 0000 e03f 0000 e03f

Second 20 bytes:

Integer: 0100 0000
Double: ff7f 0000 0000 0000
Double: 0000 e03f 0000 0000

Now basically, I'm not fully sure why the doubles here differ.

dreamcrash · Accepted Answer · 2020-12-13T21:44:29.487

Instead of:

MPI_Aint disp[3] = {0, 4, 12};

do the following:

disp[0] = offsetof(test, rank);
disp[1] = offsetof(test, begin);
disp[2] = offsetof(test, end);

do not forget to include "#include <stddef.h>", and to adapt the offset (i.e., offset = sizeof(test) * world_rank;)

Do not try to manually hardcode the offsets, it is better to use offsetof from <stddef.h> to figure it out for you. There is no guarantee that the padding used inside structures will match the values that you have hardcoded in your array for the displacements (i.e., disp)

Structure padding is a concept in C that adds the one or more empty bytes between the memory addresses to align the data in memory. (source). To a better understanding of padding look into this SO thread

MPI_File_write_at : Writing the same struct twice results in slightly different data blocks in binary file

1 Answers1