Sending java Objects over MPI

Question

My story

I started writing an mpi-program in mpiJava and managed to write a working program, but for one or another reason it doesn't want to run on the platform I need it to run on (I don't have any permissions to install anything). I'm not sure whether it was necessary (considering I get the same problems now), but I started using open-mpi with its java-bindings. The point of that is that on the platform, open-mpi is available and I thus thought it would automatically work. It didn't work and I'm still trying to find out why (it might just have something to do with the credit-system), but that's not the problem I would like to see solved here.

My problem

The real problem is that I need to send java objects over MPI and I have no idea how to do it in the right way. In mpiJava I had the advantage, there was a MPI.Object datatype available (it just serialized the objects), but open-mpi doesn't have anything like that. I already visited a post on BigInteger and a post actually providing some answers and of course the open-mpi reference, but I'm not completely sure now on what's the best approach in my cases.

I have 3 different objects I would like to send over MPI:

1. a classic BigInteger
2. a classic Object[] with 3 different objects in there
3. a less classic self-written class containing a reference to another self-written class

The self-written classes look like this:

public class AC implements Iterable<BS>, Comparable<AC>, LE, Serializable {

    private static final long serialVersionUID = -530910801257060853L;
    private static AC emptyAC = new AC();
    private static AC emptySetAC = new AC();
    static {emptySetAC.theAC.set(0);}
    
    private BitSet theAC = new BitSet();
    private BS universe = BS.universe();
    ...
    //methods
}

public class BS implements Iterable<Integer>, Comparable<BS>, Serializable {

    private static final long serialVersionUID = 4240724082500295998L;
    private static BS theEmptySet = new BS();
    private static long[] bits;

    private long theSet;
    ...
    //methods
}

The open-mpi code thus far for sending (leaving out the 'trivial' BigInteger) of my list looks like:

public static void main(String[] args) {
    
    ...
    
    //master code

    SortedMap<AC, Long> functions = new TreeMap<>();
    AC u = AC.oneSetAC(BS.universe(n));
    SortedMap<AC, BigInteger> leftIntervalSize = new TreeMap<>();
    MPI.COMM_WORLD.bcast(new Object[]{functions, leftIntervalSize, u}, 3, MPI.Object, 0);

    ...

    AC[] sendbuf = new AC[1];
    while(iterator.hasnext()) {
        MPI.COMM_WORLD.send(sendbuf, 1, MPI.OBJECT, i, 0);
    }

    ...

    //worker code
    Object[] buf = new Object[3];
    MPI.COMM_WORLD.bcast(buf, 3, MPI.OBJECT, 0);

    ...
    
    AC[] func = new AC[1];
    Status stat = MPI.COMM_WORLD.recv(func, 1, MPI.OBJECT, 0, MPI.ANY_TAG);
    
    ...
}

Note that MPI.Object is only an inheritance from my mpiJava implementation and those are the lines I need to change properly.

Proposed solutions

I already had some thoughts about it and I assume I could use the answer from the post actually providing some answers for the BigInteger if somebody could give me an example of how to use the MPI struct properly. (I really don't understand it completely from the open-mpi reference. For my own classes and the Object[] on the other hand I could do either of the following:

1. Just serialize the bloody objects.
2. Use a MPI struct (after I got how to work with it).
3. Gamble on the number of bytes needed and send them as bytes. (I would be surprised if this would work).
4. Use some other functionality that I'm about to hear about.

My question(s)

1. Could anybody give me an example on how to use the MPI struct in java? (maybe applied on one of my examples?)
2. Could anyone advise me the preferable strategy in any of the cases (and in case of serialization, point at the pitfalls)?

I'm sorry for the long post, but I prefer to be clear from the first time.

Thanks in advance

Answer 1

If you control both sides of the communication and both are in Java and you don't care too much about performance, then you can do something like this:

MyClass someObject = ...
serialized = ""
try {
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    ObjectOutputStream oos = new ObjectOutputStream(baos);
    oos.writeObject(someObject);
    oos.flush();
    serialized = baos.toString();
} 
catch (Exception e) {
    System.out.println(e);    
}
char [] message = serialized.toCharArray() ;
MPI.COMM_WORLD.Send(message, 0, message.length, MPI.CHAR, 1, 99) ;

On the other side, you just receive and basically do the same steps in reverse to deserialize.

try {
    byte b[] = serializedObject.getBytes(); 
    ByteArrayInputStream bis = new ByteArrayInputStream(b);
    ObjectInputStream ois = new ObjectInputStream(bis);
    MyClass obj = (MyClass) ois.readObject();
} catch (Exception e) {
    System.out.println(e);
}

This approach would have the advantage of being simple and get you going fast. I think it is perfectly acceptable if you don't plan on sending very many objects.

Answer 2

Eventually, I decided to serialize the Object[], using the answer of sdotdi. Because the broadcast only occurs once, this shouldn't be too bad.
For the BigInteger, I used the toByteArray() method and send the length of the array first and then the byte array itself. I suspect it to be almost as fast as using the struct of MPI, but I'm not quite sure of that. The code thus looks like:

byte[] bigintbuf = result.toByteArray();
MPI.COMM_WORLD.send(new int[]{bigintbuf.length}, 1, MPI.INT, 0, NUMTAG);
MPI.COMM_WORLD.send(bigintbuf, bigintbuf.length, MPI.BYTE, 0, 0);

For the AC class, I applied a similar trick by implementing a toLongArray() method so the code becomes:

long[] acbuf = next.toLongArray();
MPI.COMM_WORLD.send(new int[]{acbuf.length}, 1, MPI.INT, i, NUMTAG);
MPI.COMM_WORLD.send(acbuf, acbuf.length, MPI.LONG, i, 0);

It might still not be the most optimal way to do this, but I guess it approaches optimal performance rather good.