I'm trying to use MPI_Bcast
to share an instance of cudaIpcMemHandler_t, but I cannot figure out how to create the corresponding MPI_Datatype
needed for Bcast. I do not know the underlying structure of the cuda type, hence methods like this one don't seem to work. Am I missing something ?
Asked
Active
Viewed 100 times
0

dermen
- 5,252
- 4
- 23
- 34
-
1as long as you are careful, it should be possible to send it as a C-style `char` or `unsigned char` array. You can get the size of the datatype easily enough. That type is intentionally "opaque". See [here](https://stackoverflow.com/questions/23187427/broadcasting-sharing-an-array-in-mpi) – Robert Crovella Sep 27 '22 at 19:34
-
thanks! I think I got something working ... – dermen Sep 27 '22 at 20:35
1 Answers
0
Following up from the useful comment , I have a solution that seems to work, though its only been tested in a toy program:
// Broadcast the memory handle
cudaIpcMemHandler_t memHandler[1];
if (rank==0){
// set the handler content, e.g. call cudpaIpcGetMemHandle
}
MPI_Barrier(MPI_COMM_WORLD);
// share the size of the handler with other objects
int hand_size[1];
if (rank==0)
hand_size[0]= sizeof(memHand[0]);
MPI_Bcast(&hand_size[0], 1, MPI_INT, 0, MPI_COMM_WORLD);
// broadcase the handler as byte array
char memHand_C[hand_size[0]];
if (rank==0)
memcpy(&memHand_C, &memHand[0], hand_size[0]);
MPI_Bcast(&memHand_C, hand_size[0], MPI_BYTE, 0, MPI_COMM_WORLD);
if (rank >0)
memcpy(&memHand[0], &memHand_C, hand_size[0]);

dermen
- 5,252
- 4
- 23
- 34