Create MPI reduction operator from c++ lambda function

Question

I would like to write a function that wraps MPI_Allreduce, and which accepts any binary operator (as std::reduce) to be used as reduction operator by MPI. Especially, the user of such a function could use a lambda.

The following simple sample code illustrates that:

#include <mpi.h>
#include <iostream>
#include <functional>

template<typename BinaryOp>
void reduce(double *data, int len, BinaryOp op) {

  auto lambda=[op](void *a, void *b, int *len, MPI_Datatype *){
                double *aa=static_cast<double *>(a);
                double *bb=static_cast<double *>(bb);
                for (int i=0; i<*len; ++i) {
                  bb[i]=op(aa[i], bb[i]);
                }
              };


  // MPI_User_function is a typedef to: void (MPI_User_function) ( void * a, void * b, int * len, MPI_Datatype * )
  MPI_User_function *opPtr=/* black magic code that get the function pointer from the lambda */;
  MPI_Op mpiOp;
  MPI_Op_create(*opPtr, 1, &mpiOp);
  MPI_Allreduce(MPI_IN_PLACE, data, len, MPI_DOUBLE, mpiOp, MPI_COMM_WORLD);
  MPI_Op_free(&mpiOp);
}

int main() {

  MPI_Init(nullptr, nullptr);

  double data[4]={1.,2.,3.,4.};

  reduce(data, 4, [](double a, double b){return a+b;});

  int pRank;
  MPI_Comm_rank(MPI_COMM_WORLD, &pRank);
  if (pRank==0) {
    for (int i=0; i<4; ++i) {
      std::cout << data[i] << " ";
    }
    std::cout << std::endl;
  }

  MPI_Finalize();

  return 1;
}

The missing part is the code that get a function pointer from the lambda in the reduce function. From several related questions, this problem of getting a function pointer from a capturing lambda seems to be tricky but possible to solve. But I failed to have something working on this simple code (I tried some tricks with std::function, std::bind, storage of the lambda in a static variable)... So a little help would be great!

---

EDIT: Following @noma answer, I tried the following simplified code without MPI in goldbolt

#include <iostream>
#include <functional>

typedef double MPI_Datatype;

template<typename BinaryOp, BinaryOp op> // older standards
void non_lambda(void *a, void *b, int *len, MPI_Datatype *)
{}


template<typename BinaryOp>
void reduce(double *data, int len, BinaryOp op) {

  typedef void (MPI_User_function) ( void * a, void * b, int * len, MPI_Datatype * );
  MPI_User_function *opPtr = &non_lambda<decltype(+op), +op>; // older standards;
}

int main() {

  double data[4]={1.,2.,3.,4.};  
  reduce(data, 4, [](double a, double b){return a+b;});

  return 1;
}

It compile on some compilers. Here are the results:

• icc >= 19.0.1 (with -std=c++17) : OK
• clang++ >= 5.0.0 (with --std=c++17): OK
• clang++ 10.0.0 (with --std=c++14): NOK
• g++ 9.3 (with --std=c++17): NOK
• icc >= 19.0.0 (with -std=c++17) : NOK

The error message with icc 19.0.0 with -std=c++17 (or icc 19.0.1 with -std=c++14) is interesting:

   <source>(15): error: expression must have a constant value

MPI_User_function *opPtr = &non_lambda<decltype(+op), +op>; // older standards;

                                                      ^

      detected during instantiation of "void reduce(double *, int, BinaryOp) [with BinaryOp=lambda [](double, double)->double]" at line 21               

And indeed, I don't really understand the passing of the 'op' variable which is a runtime argument of the function reduce as the second template parameter of the non_lambda function... Is it an obscure c++17 functionality that only some of the compilers support?

Answer 1

I think the lambda approach is not possible here as it is a capturing lambda, see https://stackoverflow.com/a/28746827/7678171

We can use a function template with the BinaryOp as a template value parameter instead of a Lambda here. This assumes that the BinaryOp is either a function pointer, or a capture-less lambda, that can be converted into one. Instead of the lambda inside your reduce we introduce:

template<auto op> // this is C++17, so use --std=c++17
// template<typename BinaryOp, BinaryOp op> // older standards
void non_lambda(void *a, void *b, int *len, MPI_Datatype *)
{
    double *aa=static_cast<double *>(a);
    double *bb=static_cast<double *>(bb);
    for (int i=0; i<*len; ++i) {
        bb[i]=op(aa[i], bb[i]);
    }
}

The Black Magic line then is:

/* black magic code that get the function pointer from the lambda */
MPI_User_function *opPtr = &non_lambda<+op>; // NOTE: the + implies the lamda to function pointer conversion here
// MPI_User_function *opPtr = &non_lambda<decltype(+op), +op>; // older standards;

Hope this helps.

---

NOTE: I got this compiled using Clang 6.0, but g++ 7.5 failed (possible compiler bug?):

error: no matches converting function ‘non_lambda’ to type ‘void (*)(void*, void*, int*, struct ompi_datatype_t**)’
   MPI_User_function *opPtr = &non_lambda<+op>;
                      ^~~~~
note: candidate is: template<auto op> void non_lambda(void*, void*, int*, ompi_datatype_t**)
 void non_lambda(void *a, void *b, int *len, MPI_Datatype *)

Maybe newer g++ versions work.