Allgathering or Scattering a chunk of a multi-dim array in FORTRAN

Question

Following @Jonathan's comment, I am trying to reframe my question. I have a five-dim array which is calculated in parts at various processors. I need to allgather (in the MPI terminology) the elements at all the processors after the chunks have been calculated.

Following Jonathan's answer (https://stackoverflow.com/a/17530368/2895678) to a similar question on a 2D array I have built my test program, which successfully scatters (via MPI_Scatterv) the chunks. The code is

program scatter
  use mpi
  implicit none

  integer, dimension(2,2,2,6,2) :: gpsi
  integer, dimension(2,2,2,2,2) :: local
  integer, dimension(3) :: displs,counts
  integer :: ierr,rank,comsize
  integer :: p,newtype,i,j,k,l,m,intsize,resizedtype
  integer, dimension(5) :: totsize,subsize,starts
  integer, dimension(MPI_STATUS_SIZE) :: rstatus
  integer(kind=MPI_ADDRESS_KIND) :: extent, begin

  call MPI_Init(ierr)
  call MPI_Comm_size(MPI_COMM_WORLD, comsize, ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)

  gpsi = 0.0
  if(rank == 0) then
    call f(gpsi)
    do m = 1,2
      do l = 1,6
        do k = 1,2
          do j = 1,2
            do i = 1,2
              print*,i,j,k,l,m,gpsi(i,j,k,l,m)
            enddo
          enddo
        enddo
      enddo
    enddo
  endif

  totsize = [2,2,2,6,2]
  subsize = [2,2,2,2,2]
  starts = [0,0,0,0,0]

  call MPI_type_create_subarray(5,totsize,subsize,starts, &
                                MPI_ORDER_FORTRAN, MPI_INTEGER, &
                                newtype,ierr)
  call MPI_type_size(MPI_INTEGER,intsize,ierr)
  extent = intsize
  begin = 0
  call MPI_type_create_resized(newtype,begin,extent,resizedtype,ierr)
  call MPI_type_commit(resizedtype,ierr)
  counts = 1
  displs = [0,16,32]
  call MPI_scatterv(gpsi,counts,displs,resizedtype,local,32, &
                  MPI_INTEGER,0,MPI_COMM_WORLD,ierr)

  do p = 1,comsize
    if(rank == p-1) then
       print*,'Rank =',rank
       print*,local(:,:,:,:,:)
       print*,''
    endif
    call MPI_barrier(MPI_COMM_WORLD,ierr)
  enddo

  call MPI_Type_free(newtype,ierr)
  call MPI_Finalize(ierr)
end program scatter

subroutine f(psi)
  implicit none
  integer, dimension(96) :: psi
  integer :: i

  do i = 1,96
    psi(i) = i
  enddo
  return
end

Here, I am sending the chunks gpsi(:,:,:,1:2,:) to proc 1 (i.e. root), gpsi(:,:,:,3:4,:) to proc 2, and gpsi(:,:,:,5:6,:) to proc 3 (for three procs). Note that I have created a dummy subroutine called f(psi) to allocate all the 96 elements of the multi-dim array gpsi in a contiguous memory space so that I can correctly calculate the displacement array displs, which in this case is displs[0,16,32]. In this case, I successfully have the values 1,2,…,15,16 and 49,50,…,63,64 in proc 1, 17,18,…,31,32 and 65,66,…,79,80 in proc 2, and 33,34,…,47,48 and 81,82,…,95,96 in proc 3 after MPI_Scatterv.

My actual problem is however, where I would like to scatter dissimilar chunks and all gather them afterwards. For example, if I want to send gpsi(:,:,:,1:2,:) to proc 1, gpsi(:,:,:,3,:) to proc 2, and say gpsi(:,:,:,4:6,:) to proc 3. The question is how to construct the subarrays now as each subarray is of different size so, in principle, the new data type for each processor is now different. How to achieve that? Will highly appreciate any help.

Regards,

Madhurjya

Please post a comment on the linked answer for clarifications instead of asking a new question! — Alexander Vogt, Jul 11 '14 at 07:28
If you edit this question into a more specific question about calculating the extents in your case it would work better for SO. In general, you just map out how data is laid out in memory, and count to find (a) where the data pieces begin/end, and (b) what is a useful extent to use to express that. In the case of the 3x3 subarrays of a 6x6 array, the subarrays began at the 1st, 4th (=1+3), 19th (=3*6+1), and 22st (3*6+3+1) element of the large array - or, in the C-style 0-based indexing MPI uses, [0,3,18,21]. Given that, I chose 3 for the extent of the data type, but 1 would work as well. — Jonathan Dursi, Jul 11 '14 at 15:42
Thanks @JonathanDursi! I got it working for a FORTRAN multi-dim array. However, if I want to scatterv different amount of data, then how should I proceed. My test array is gpsi(2,3,4,6,2), a 5-dim array and I have scattered successfully gpsi(:,:,:,1:2,:), gpsi(:,:,:,3:4,:), and gpsi(:,:,:,5:6,:) among three processors just as you have done in your example. How do I do it if I want to scatter say gpsi(:,:,:,1:2,:), gpsi(:,:,:,1,:), and gpsi(:,:,:,4:6,:) among three procs? The local array must be of different size in this case but MPI_Scatterv uses the local array which is seen by all procs. — Madhurjya, Jul 17 '14 at 02:39
Hi, @Madhurjya - it's a bit too complicated to answer in comments. Why don't you edit this question to reflect your question in comments above, include some code that you're having trouble with, and we can re-open the question and give you a more detailed answer. — Jonathan Dursi, Jul 17 '14 at 13:45

Allgathering or Scattering a chunk of a multi-dim array in FORTRAN

0 Answers0