2

I am trying to build a distributed application with C++11 and MPI. I would like to send a lambda closure, to another computer on my cluster over a MPI_send() call.

For example, say I had the following lambda

auto x = [x, y] () { return x + y; };

I would like to define a function, say pack() so that I could call the MPI send -

MPI_Send(pack(f_xy), SOMESIZE, MPI_CHAR, 1, 0, MPI_COMM_WORLD);

Could someone tell me how I could define the pack function (Also an unpack function at the receivers end?). I have been looking around online, but have not been successful in finding any way to do this.

ssb
  • 7,422
  • 10
  • 36
  • 61
  • There really isn't a way to serialize functions in general or lambdas in particular in C++ - eg, [this question](http://stackoverflow.com/questions/7771842/serialize-c-functor) or [this one](http://stackoverflow.com/questions/12338265/serializing-function-objects). – Jonathan Dursi Oct 15 '14 at 17:01
  • @JonathanDursi an [answer](http://stackoverflow.com/a/22772214/1328439) to the latter question seems to suggest that closures can in fact be serialized. – Dima Chubarov Oct 17 '14 at 06:19
  • 1
    @DmitriChubarov - as best as I can tell, that answer seems to rely on serializing function pointers or implementation-dependent opaque objects representing which lambda to execute. But that's not the function itself, and there's no obvious robust way to convert either to the relevant function even if that function already exists in the target process' memory. – Jonathan Dursi Oct 17 '14 at 11:36

1 Answers1

1

I can think of three approaches, neither easy or good.

The first is to compile a dll and serialize it and send it, write it out at the other end and load and execute it. This requires interacting with your build system. It works far better for function objects than lambdas.

The second involves using something like boost phoenix to write pseudo-C++ in a way that the parse tree is visible from C++. Then send said parse tree with bindings over the wire, and interpret it at the other end.

The third is to attach a scripting engine to both ends, and send a script (again with bindings and bound data) over the wire.

None are easy, and none are actually sending a lambda over the wire. In general, lambdas are types more than they are data, and MPI does not support transmitting types.

Yakk - Adam Nevraumont
  • 262,606
  • 27
  • 330
  • 524
  • 1
    Well, it's not that _MPI_ supports or doesn't support transmitting types or functions; MPI is just putting ones and zeros on the wire and doesn't really care what they are other than some semantic sugar to make the programmer's job easier. It's that C++ (say) doesn't have a way of serializing or unserializing types or functions or what have you. With Python and MPI, for instance, one can send serialized (pickled) Python functions or objects quite easily; but that's not how C++ or similar languages work. – Jonathan Dursi Oct 15 '14 at 18:03
  • @JonathanDursi ok: MPI C++ wrappers don't support sending functions. You could have a library that serialized actual functions between sufficiently similar systems. It would be an extension, like `fork()`. Heck, you could implement it as `fork()`, followed by a core dump of the child process, which you send over the wire and run from the core dump. But that would be silly. – Yakk - Adam Nevraumont Oct 15 '14 at 20:02
  • Do you have any examples of the boost phoenix libraries doing something like this? – ssb Oct 16 '14 at 07:21
  • @subzero nope. The other end would be interesting, because it couldn't be boost pheonix like. But because you have access to the parse tree on the sending end, it can be serialized. Deserialization cannot be to the same type, as type cannot be determined by data-on-wire, and instead you'd have to basically write an interpreter. – Yakk - Adam Nevraumont Oct 16 '14 at 14:07