sending c struct via MPI fails partially

Question

I am sending a (particle) struct using the MPI_Type_create_struct() as done e.g. here, or explained in detail here. I'm collecting all particles which are going to a specific proc, memcpy() them into the send buffer and MPI_Isend() them.

So far, so good. MPI_Iprob()'ing for the message gives me the right count of particles sent. So I MPI_Recv() the buffer and extract the data (now even by copying the struct one by one). No matter how many particles I send, only the first particles' data are correct.

There are three possible mistakes:

1. The MPI_Type_create_struct() doesn't create a proper map of my struct, due to my usage of offset of() like in the first link. Maybe my struct contains a non visible padding as explained in the second link.
2. I'm doing some simple mistakes while copying particles into the send buffer and from the receive buffer back (I do print the send buffer - and it works - but maybe I'm overlooking something)
3. Something totally different.

(sorry for the really ugly presentation of the code, I could not manage to present it in a descent way. You'll find the code here - the line is already marked - on Github, too!)

Here are the construction of the mpi datatype,

typedef struct {
        int                 ID;
        double              x[DIM];
} pchase_particle_t;

 const int           items = 2;
 int                 block_lengths[2] = {1, DIM};
 MPI_Datatype        mpi_types[2] = {MPI_INT, MPI_DOUBLE};
 MPI_Aint            offsets[2];
 offsets[0] = offsetof(pchase_particle_t, ID);
 offsets[1] = offsetof(pchase_particle_t, x);
 MPI_Type_create_struct(items, block_lengths, offsets, mpi_types, &W->MPI_Particle);
 MPI_Type_commit(&W->MPI_Particle);

the sending

/* handle all mpi send/recv status data */
MPI_Request        *send_request = P4EST_ALLOC(MPI_Request, W->p4est->mpisize);
MPI_Status         *recv_status = P4EST_ALLOC(MPI_Status, W->p4est->mpisize);
/* setup send/recv buffers */
pchase_particle_t **recv_buf = P4EST_ALLOC(pchase_particle_t *, num_senders);
pchase_particle_t **send_buf = P4EST_ALLOC(pchase_particle_t *, num_receivers);
int                 recv_count = 0, recv_length, flag, j;

/* send all particles to their belonging procs */
for (i = 0; i < num_receivers; i++) {
  /* resolve particle list for proc i */
  sc_list_t          *tmpList = *((sc_list_t **) sc_array_index(W->particles_to, receivers[i]));
  pchase_particle_t * tmpParticle;
  int                 send_count = 0;

  /* get space for the particles to be sent */
  send_buf[i] = P4EST_ALLOC(pchase_particle_t, tmpList->elem_count);

  /* copy all particles into the send buffer and remove them from this proc */
  while(tmpList->first != NULL){
    tmpParticle = sc_list_pop(tmpList);
    memcpy(send_buf[i] + send_count * sizeof(pchase_particle_t), tmpParticle, sizeof(pchase_particle_t));
    /* free particle */
    P4EST_FREE(tmpParticle);
    /* update particle counter */ 
    send_count++;
  }

  /* print send buffer */
  for (j = 0; j < send_count; j++) {
    pchase_particle_t  *tmpParticle = send_buf[i] + j * sizeof(pchase_particle_t);
    printf("[pchase %i sending] particle[%i](%lf,%lf)\n", W->p4est->mpirank, tmpParticle->ID, tmpParticle->x[0], tmpParticle->x[1]);
  }

  printf("[pchase %i sending] particle count: %i\n", W->p4est->mpirank, send_count);
  /* send particles to right owner */
  mpiret = MPI_Isend(send_buf[i], send_count, W->MPI_Particle, receivers[i], 13, W->p4est->mpicomm, &send_request[i]);
  SC_CHECK_MPI(mpiret);
}

and the receiving.

recv_count = 0;
/* check for messages until all arrived */
while (recv_count < num_senders) {
  /* probe if any of the sender has already sent his message */
  for (i = 0; i < num_senders; i++) {
    MPI_Iprobe(senders[i], MPI_ANY_TAG, W->p4est->mpicomm,
        &flag, &recv_status[i]);
    if (flag) {
      /* resolve number of particles receiving */
      MPI_Get_count(&recv_status[i], W->MPI_Particle, &recv_length);
      printf("[pchase %i receiving message] %i particles arrived from sender %i with tag %i\n",
          W->p4est->mpirank, recv_length, recv_status[i].MPI_SOURCE, recv_status[i].MPI_TAG);
      /* get space for the particles to be sent */
      recv_buf[recv_count] = P4EST_ALLOC(pchase_particle_t, recv_length);
      /* receive a list with recv_length particles */ 
      mpiret = MPI_Recv(recv_buf[recv_count], recv_length, W->MPI_Particle, recv_status[i].MPI_SOURCE,
          recv_status[i].MPI_TAG, W->p4est->mpicomm, &recv_status[i]);
      SC_CHECK_MPI(mpiret);

      /*
       * insert all received particles into the
       * push list
       */
      pchase_particle_t  *tmpParticle;
      for (j = 0; j < recv_length; j++) {
        /*
         * retrieve all particle details from
         * recv_buf
         */
        tmpParticle = recv_buf[recv_count] + j * sizeof(pchase_particle_t);
        pchase_particle_t *addParticle = P4EST_ALLOC(pchase_particle_t,1);
        addParticle->ID=tmpParticle->ID;
        addParticle->x[0] = tmpParticle->x[0];
        addParticle->x[1] = tmpParticle->x[1];

        printf("[pchase %i receiving] particle[%i](%lf,%lf)\n",
            W->p4est->mpirank, addParticle->ID, addParticle->x[0], addParticle->x[1]);
        /* push received particle to push list and update world counter */
        sc_list_append(W->particle_push_list, addParticle);
        W->n_particles++;
      }
      /* we received another particle list */
      recv_count++;
    }
  }
}

edit: reindented.. edit: Only the first particles' data is correct, means that all it's properties (ID and coordinates) are identical to that of the sent particle. The others however are initialized with zeros i.e. ID=0, x[0]=0.0, x[1]=0.0. Maybe that's a hint for the solution.

Answer 1

There is an error in your pointer arithmetic. send_buf[i] is already of type pchase_particle_t * and therefore send_buf[i] + j * sizeof(pchase_particle_t) does not point to the j-th element of the i-th buffer but rather to the j * sizeof(pchase_particle_t)-th element. Thus your particles are not stored contiguously in memory but rather separated by sizeof(pchase_particle_t) - 1 empty array elements. These get sent instead of the correct particles because the MPI_Send call accesses buffer memory contiguously. The same applies to the code of the receiver.

You do not see the error in the sender code because your debug print uses the same incorrect pointer arithmetic and hence accesses memory using the same stride. I guess your send counts are small and you get memory allocated on the data segment heap, otherwise you should have received SIGSEGV for out-of-bound array access very early in the data packing process (e.g. in the memcpy part).

Resolution: do not multiply the array index by sizeof(pchase_particle_t).