0

I have that each processor has its own unique matrix, A, of size Nx2 where N varies with processor. I want to collect all these matrices into one single buff (NxP)x2 matrix, where P is the number of processors.

Size wise in Fortran they are allocated like, 

A(N,2) 
buff(N*P,2)

As an example, let P = 2 and the A matrices for each processor be,

for Proc-1
10  11
10  11

for Proc-2
20  21
20  21

To this end I use MPI_GATHERV and save the individual matrices in the buff matrix. If I do this then buff will look like this, 

10  20
10  20
11  21
11  21

But what I want is the matrix to look like this, 

10  11
10  11
20  21
20  21

In memory (I think) Buff : |10 , 10 , 20, 20 , 11 , 11 , 21 , 21|

Sample code is below,

...
! size = 2
root = 0
ALLOCATE ( count(size), num(size) )
! -----------------------------------------------------------
! Mock data 
! -----------------------------------------------------------
IF(rank.eq.0) THEN 
    m = 2
    mm = m*2
    allocate(A(m,2))
    A(1,1) = 10
    A(1,2) = 11
    A(2,1) = 10
    A(2,2) = 11
ELSE
    m = 2
    mm = m*2
    allocate(A(m,2))
    A(1,1) = 20
    A(1,2) = 21
    A(2,1) = 20
    A(2,2) = 21
END IF

! -----------------------------------------------------------
! send number of elements
! -----------------------------------------------------------
CALL MPI_GATHER(mm,1,MPI_INTEGER,count,1,MPI_INTEGER,root,cworld,ierr)

! -----------------------------------------------------------
! Figure out displacement vector needed for gatherv
! -----------------------------------------------------------
if(rank.eq.0) THEN
    ALLOCATE (buff(SUM(count)/2,2), disp(size), rdisp(size))
    rdisp = count
    disp(1) = 0
    DO i = 2,size
        disp(i) = disp(i-1) + count(i-1) 
    END DO
END IF

! -----------------------------------------------------------
! Rank-0 gathers msg 
! -----------------------------------------------------------
CALL MPI_GATHERV(A,mm,MPI_INTEGER,buff,rdisp,disp,MPI_INTEGER,root,cworld,ierr)

! -----------------------------------------------------------
! Print buff
! -----------------------------------------------------------
if(rank.eq.0) THEN
    DO i = 1,sum(count)/2
        print*, buff(i,:)
    end do
END IF

I have looked at Using Gatherv for 2d Arrays in Fortran but am a little confused with the explanation.

I’m not very familiar with the MPI details, but is there a "simple" way to gather all the matrices and place them in the correct memory position in buff?

**** Edit ****

Fallowing what Gilles Gouaillardet suggested. I'm trying to figure how to do that,

The derived type for sending the rows should look something like this (I think),

CALL MPI_TYPE_vector(2,1,2,MPI_INTEGER,MPI_ROWS,ierr)
CALL MPI_TYPE_COMMIT(MPI_ROWS,ierr)

Then I extend,

call MPI_Type_size(MPI_INTEGER, msg_size, ierr)
lb = 0
extent = 2*msg_size
call MPI_Type_create_resized(MPI_ROWS, lb, extent , MPI_ROWS_extend,   ierr)
CALL MPI_TYPE_COMMIT(, MPI_ROWS_extend,ierr)

I’m trying to understand why I need the second derived type for receiving. I’m not sure how that one should look like.

Michael
  • 41,989
  • 11
  • 82
  • 128
Al007
  • 39
  • 4
  • You can build two derived datatypes for the rows (one for sending, the other for receiving) and then `MPI_Gatherv()`. Note you will have to `MPI_Type_create_resized()` the derived datatypes. – Gilles Gouaillardet Apr 07 '18 at 00:18
  • I'm not sure what you mean by creating one derived datatype for sending and receiving. By receiving you mean a derived datatype defined on root only, could you expand a bit on your suggestion? – Al007 Apr 11 '18 at 04:47
  • strictly speaking, you need the derived datatype for sending on all the ranks, and the derived datatype for receiving only on root. – Gilles Gouaillardet Apr 11 '18 at 04:51
  • I edited my original post for my fallow-up question of your suggestion – Al007 Apr 11 '18 at 20:55
  • your derived datatype is incorrect. try to send three rows of a 3x2 matrix and receive 6 `MPI_INT`, and check the result. – Gilles Gouaillardet Apr 11 '18 at 23:39

0 Answers0