0

I'm trying to use MPI_Bcast to share an instance of cudaIpcMemHandler_t, but I cannot figure out how to create the corresponding MPI_Datatype needed for Bcast. I do not know the underlying structure of the cuda type, hence methods like this one don't seem to work. Am I missing something ?

dermen
  • 5,252
  • 4
  • 23
  • 34
  • 1
    as long as you are careful, it should be possible to send it as a C-style `char` or `unsigned char` array. You can get the size of the datatype easily enough. That type is intentionally "opaque". See [here](https://stackoverflow.com/questions/23187427/broadcasting-sharing-an-array-in-mpi) – Robert Crovella Sep 27 '22 at 19:34
  • thanks! I think I got something working ... – dermen Sep 27 '22 at 20:35

1 Answers1

0

Following up from the useful comment , I have a solution that seems to work, though its only been tested in a toy program:

//  Broadcast the memory handle

    cudaIpcMemHandler_t  memHandler[1];
    if (rank==0){
        // set the handler content, e.g. call cudpaIpcGetMemHandle
    }
    MPI_Barrier(MPI_COMM_WORLD);
    
    // share the size of the handler with other objects
    int hand_size[1];
    if (rank==0)
        hand_size[0]= sizeof(memHand[0]);
    MPI_Bcast(&hand_size[0], 1, MPI_INT, 0, MPI_COMM_WORLD);

    // broadcase the handler as byte array
    char memHand_C[hand_size[0]];
    if (rank==0)
        memcpy(&memHand_C, &memHand[0], hand_size[0]);
    MPI_Bcast(&memHand_C, hand_size[0], MPI_BYTE, 0, MPI_COMM_WORLD);
    if (rank >0)
        memcpy(&memHand[0], &memHand_C, hand_size[0]);
dermen
  • 5,252
  • 4
  • 23
  • 34