How do I correctly set the "send counts" and "displacement" arguments in MPI_scatterv?

Question

I am trying to split up a 2 dimensional array's rows between "n" processes using MPI_Scatterv. The two arguments that are stopping me is "send_counts" and "displacements". I know the textbook definition of what these arrays do, but I need a way to dynamically create these arrays to accept any length 2 dimensional array, especially rows of a 2D array that are not evenly divisible by the number of processes.

The inspiration of this approach comes from here (building the send_counts and displacement array): https://gist.github.com/ehamberg/1263868 I understand the approach, but I wonder if this implementation only works for even 2D arrays (matrices).

questions: Could the problem have something to do with the 2D array not being contiguous?

Are the correct displacements concerning the memory blocks of the data type in question(ie should my displacements be 4 because floats are 4 bytes of memory?)

#include <iostream>
#include <fstream>
#include <sstream>
#include "mpi.h"
#include <stdio.h>


#define ROW 75 
#define COL 5

void importData(std::string str, float (*dest)[75][5], int length) {

std::ifstream infile(str);

int i = 0;
int j = 0;

std::string a;

while (getline(infile, a)) {

    std::stringstream ss(a);
    std::string token;
    i = 0;


    while (getline(ss, token, ',')) {

        if (i < length) {

            (*dest)[i][j] = strtof(token.c_str(), NULL);
        }

        else {

            i++;
        }

        j++;

    }


 }


}


int main(int argc, char **argv)
{

float iris[75][5] = { {} };

importData("Iris.test", &iris, 5);


int rank, comm_sz;

int sum = 0;

int rem = (ROW*COL) % comm_sz;

int * send_counts;
int * displs;


MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);


int row[1000];




send_counts = (int *)malloc(sizeof(float)*comm_sz);
displs = (int *)malloc(sizeof(float)*comm_sz);


// calculate send counts and displacements
for (int i = 0; i < comm_sz; i++) {
    send_counts[i] = (ROW*ROW) / comm_sz;
    if (rem > 0) {
        send_counts[i]++;
        rem--;
    }

    displs[i] = sum;
    sum += send_counts[i];
}



if (rank == 0){


}


// Scatter the big table to everybody's little table, scattering the rows
MPI_Scatterv(iris, send_counts, displs, MPI_FLOAT, row, 100, MPI_FLOAT, 0, 
MPI_COMM_WORLD);
//                              displacements      recv buffer, recv count
std::cout << "%d: " << rank << std::endl;

for (int i = 0; i < send_counts[rank]; i++) {
    std::cout << "%f\t" << row[i] << std::endl;
}

MPI_Finalize();



}

I expect each of "n" processes to print out a portion of the rows of the passed array.

This is the error I get:

An error occurred in MPI_Scatterv reported by process [2187067393,0] on communicator MPI_COMM_WORLD MPI_ERR_TRUNCATE: message truncated MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, and potentially your MPI job)

**Note: the data file is 75 lines with 5 float numbers on each line, comma delimited

Answer 1

The issue is you have non-matching signatures between what you send (e.g. sendcounts and sendtype) and what you receive (e.g. recvcount and recvtype).

In your case, since you receive (hard coded) 100 MPI_FLOAT and you send MPI_FLOAT, it can only work if all send_counts[i] == 100.

I guess the right fix is to change value of recvcount. On rank i, it should have the same value than send_counts[i] on the root rank (e.g. rank 0 in your case)