3

I'm having a segmentation fault that I can not really understand in a simple code, that just:

  • calls the MPI_INIT
  • duplicates the global communicator, via MPI_COMM_DUP
  • creates a group with half of processes of the global communicator, via MPI_COMM_GROUP
  • finally from this group creates a new communicator via MPI_COMM_CREATE_GROUP

Specifically I use this last call, instead of just using MPI_COMM_CREATE, because it's only collective over the group of processes contained in group, while MPI_COMM_CREATE is collective over every process in COMM. The code is the following

program mpi_comm_create_grp
  use mpi
  IMPLICIT NONE

  INTEGER :: mpi_size,  mpi_err_code
  INTEGER :: my_comm_dup, mpi_new_comm, mpi_group_world, mpi_new_group
  INTEGER :: rank_index
  INTEGER, DIMENSION(:), ALLOCATABLE :: rank_vec

  CALL mpi_init(mpi_err_code)
  CALL mpi_comm_size(mpi_comm_world, mpi_size, mpi_err_code)

  !! allocate and fill the vector for the new group
  allocate(rank_vec(mpi_size/2))
  rank_vec(:) = (/ (rank_index , rank_index=0, mpi_size/2) /)

  !! create the group directly from the comm_world: this way works
  ! CALL mpi_comm_group(mpi_comm_world, mpi_group_world, mpi_err_code)

  !! duplicating the comm_world creating the group form the dup: this ways fails
  CALL mpi_comm_dup(mpi_comm_world, my_comm_dup, mpi_err_code)
  !! creatig the group of all processes from the duplicated comm_world
  CALL mpi_comm_group(my_comm_dup, mpi_group_world, mpi_err_code)

  !! create a new group with just half of processes in comm_world
  CALL mpi_group_incl(mpi_group_world, mpi_size/2, rank_vec,mpi_new_group, mpi_err_code)

  !! create a new comm from the comm_world using the new group created
  CALL mpi_comm_create_group(mpi_comm_world, mpi_new_group, 0, mpi_new_comm, mpi_err_code)

  !! deallocate and finalize mpi
  if(ALLOCATED(rank_vec)) DEALLOCATE(rank_vec)
  CALL mpi_finalize(mpi_err_code)
end program !mpi_comm_create_grp

If instead of duplicating the COMM_WORLD, I directly create the group from the global communicator (commented line), everything works just fine.

The parallel debugger I'm using traces back the seg fault to a call to MPI_GROUP_TRANSLATE_RANKS, but, as far as I know, the MPI_COMM_DUP duplicates all the attributes of the copied communicator, ranks numbering included.

I am using the ifort version 18.0.5, but I also tried with the 17.0.4, and 19.0.2 with no better results.

Sparonuz
  • 145
  • 7
  • what if you call `MPI_Comm_create_group()` only on ranks `0` to `mpi_size/2 - 1` ? This is your rationale for using `MPI_Comm_create_group()` after all ... – Gilles Gouaillardet Mar 20 '19 at 00:30
  • 2
    I was able to work around this issue by simply invoking `MPI_Comm_create_group()` on `my_comm_dup` instead of `MPI_COMM_WORLD` – Gilles Gouaillardet Mar 20 '19 at 00:52
  • Why aren't you using MPI_COMM_SPLIT which is much easier for this sort of thing? MPI_COMM_DUP is already synchronising over the "parent" communicator, so it can't be for that reason – Ian Bush Mar 20 '19 at 06:40
  • @GillesGouaillardet I knew that using my_comm_dup instead of the comm_world would work. Unfortunately, in the real program I am developing, between the duplication and the mpi_comm_create the my_comm_dup changes (basically the mpi_comm_dup = mpi_comm_new), so I need to use the original comm_world, basically to recreate the old my_comm_dup. This was just a simplified example. – Sparonuz Mar 20 '19 at 09:13
  • @IanBush Beside the fact that I don't want to split a communicator, but just to create a group from that, there is the reason I already pointed out for not using MPI_COMM_CREATE: these are blocking function, and, in my program, not all processes are going to execute this call. – Sparonuz Mar 20 '19 at 09:26
  • MPI_COMM_DUP is blocking - all communicator creators are blocking. See https://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node102.html and in particular "The following are collective functions that are invoked by all processes in the group associated with comm. " Or are you only concerned about blocking after duplicating the communicator? – Ian Bush Mar 20 '19 at 09:52
  • @IanBush What I ment is that `MPI_COMM_CREATE_GROUP` is blocking just between the processes in the group you use for creating the new communicator, so basically it's non blocking, even being a function that creates a communicator. `MPI_COMM_CREATE, DUP ` and `SPLIT` are blocking with respect with all the processes in the parent communicator, so are basically blocking. And of course you're right, I was not clear, my concern for the barriers is just after the duplication. – Sparonuz Mar 20 '19 at 10:38
  • 1
    Open MPI and MPICH 3.3 are working just fine, so this is something you have to report to Intel. (IntelMPI is proprietary software based on MPICH). A temporary workaround is then to use a MPI library that *works for you* – Gilles Gouaillardet Mar 21 '19 at 08:36
  • Yes, I figured out the same, I already did a report to Intel ... Do you think you can make your comment an answer so I can mark the question as resolved? Thank you – Sparonuz Mar 22 '19 at 08:14
  • @GillesGouaillardet I think we were both wrong. Thake a look at my answer if you like! – Sparonuz Apr 02 '19 at 07:13

2 Answers2

3

Well the thing is a little tricky, at least for me, but after some tests and help, the root of the problem was found.

In the code

CALL mpi_comm_create_group(mpi_comm_world, mpi_new_group, 0, mpi_new_comm, mpi_err_code)

Creates a new communicator for the group mpi_new_group, previously created. However the mpi_comm_world, which is used as first argument, is not in the same context as mpi_new_group, as explained in the mpich reference:

MPI_COMM_DUP will create a new communicator over the same group as comm but with a new context

So the correct call would be: 

CALL mpi_comm_create_group(my_comm_copy, mpi_new_group, 0, mpi_new_comm, mpi_err_code)

I.e. , replacing the mpi_comm_world for my_comm_copy, that is the one from which the mpi_group_world was created.

I am still not sure why it is working with OpenMPI, but it is generally more tolerant with this sort of things.

Sparonuz
  • 145
  • 7
  • thanks for the heads up ! per the standard "MPI_COMM_CREATE_GROUP must be called by all processes in group, which is a subgroup of the group of comm.". My interpretation of the standard is that `MPI_COMM_WORLD` and `my_comm_copy` have the same group, so `mpi_new_group` **is** indeed a subgroup of `MPI_COMM_WORLD`. IntelMPI (MPICH ?) interpretation looks much stricter here (e.g. `mpi_new_group` is **not** a subgroup of `MPI_COMM_WORLD`). Better ask the MPI forum imho. – Gilles Gouaillardet Apr 02 '19 at 07:34
  • as a side note, since IntelMPI (MPICH ?) is crashing instead of displaying a user friendly error message, that could indicate a bug rather than a different (incorrect ?) interpretation. – Gilles Gouaillardet Apr 02 '19 at 07:36
  • 1
    I think I'll try and ask the MPI forum then, it makes sense! I'll update the answer as soon as I've got a reply – Sparonuz Apr 02 '19 at 09:25
1

Like suggested in the comments I wrote to openmpi user list, and they replied

That is perfectly valid. The MPI processes that make up the group are all part of comm world. I would file a bug with Intel MPI.

So I try and post a question on Intel forum. It is a bug they solved in the last version of the libray, 19.3.

Sparonuz
  • 145
  • 7