Sending an Array of Strings using Message Passing Interface

Question

I would like to send an array of strings from the master to a slave thread using Messgae Passing Interface (MPI). i.e. String [] str = new String [10] str[0]= "XXX" ... etc

How can I do that while avoiding to send each of the elements in this array as a chain of characters?

I succeeded to send an array of integers in one send operation ... but I don't know how to do that when it is about an array of strings

score 3 · Answer 1 · answered Sep 23 '14 at 17:47

Sending arrays of strings, especially if of varying sizes, is quite an involving process. There are several options but the most MPI-friendly one is to use the packing and unpacking facilities of MPI, exposed in mpiJava as Comm.Pack, Comm.Unpack, and Comm.Pack_size.

You could do something of the sort:

Sender

byte[][] bytes = new byte[nStr][];
int[] lengths = new int[nStr];

int bufLen = MPI.COMM_WORLD.Pack_size(1, MPI.INT);

bufLen += MPI.COMM_WORLD.Pack_size(nStr, MPI.INT);

for (int i = 0; i < nStr; i++) {
   bytes[i] = str[i].getBytes(Charset.forName("UTF-8"));
   lengths[i] = bytes[i].length;
   bufLen += MPI.COMM_WORLD.Pack_size(lengths[i], MPI.BYTE);
}

byte[] buf = new byte[bufLen];
int position = 0;

int nStrArray[] = new int[1];
nStrArray[0] = nStr;

position = MPI.COMM_WORLD.Pack(nStrArray, 0, 1, MPI.INT,
                               buf, position);

position = MPI.COMM_WORLD.Pack(lengths, 0, nStr, MPI.INT,
                               buf, position);

for (int i = 0; i < nStr; i++) {
   position = MPI.COMM_WORLD.Pack(bytes[i], 0, lengths[i], MPI.BYTE,
                                  buf, position);
}

MPI.COMM_WORLD.Send(buf, 0, bufLen, MPI.PACKED, rank, 0);

Having string lengths in an auxiliary array and packing it at the beginning of the message simplifies the receiver logic.

Receiver

Assumes that the sender is rank 0.

Status status = MPI.COMM_WORLD.Probe(0, 0);
int bufLen = status.Get_count(MPI.PACKED);

byte[] buf = new byte[bufLen];

MPI.COMM_WORLD.Recv(buf, 0, bufLen, MPI.PACKED, status.source, status.tag);

int position = 0;

int nStrArray[] = new int[1];

position = MPI.COMM_WORLD.Unpack(buf, position,
                                 nStrArray, 0, 1, MPI.INT);

int nStr = nStrArray[0];
int lengths[] = new int[nStr];

position = MPI.COMM_WORLD.Unpack(buf, position,
                                 lengths, 0, nStr, MPI.INT);

String[] str = new String[nStr];

for (int i = 0; i < nStr; i++) {
   byte[] bytes = new byte[lengths[i]];

   position = MPI.COMM_WORLD.Unpack(buf, position,
                                    bytes, 0, lengths[i], MPI.BYTE);
   str[i] = new String(bytes, "UTF-8");
}

Disclaimer: I don't have MPJ Express installed and my Java knowledge is very limited. The code is based on the mpiJava specification, the MPJ Express JavaDocs, and some examples found on the Internet.

Oh that's really an intelligent way .. though I found a shorter answer, I learnt a nice way to pack and unpack data sent to receiver :) I will include it in my implementation :) — Study Learn, Sep 23 '14 at 19:05

Rob Latham · Accepted Answer · 2014-09-23T16:27:07.627

2

I don't know Java, but I'll give you the C answer. The concepts -- particularly the two approaches one might take to solve this - are the same in any language, though.

Imagine if this were a simple c-string (some characters terminated with '\0'). There are two approaches:

over-provision memory and receive up to some limit,
or send a message indicating how much data to expect.

Do you have a maximum length? (e.g. PATH_MAX or something like that). If you do not need every byte of memory, you could do

MPI_Send(str, strlen(str), MPI_CHAR, slave_rank, slave_tag, MPI_COMM_WORLD);

and you'd pair that with

MPI_Recv(str, MAX_LENGTH, MPI_CHAR, master_rank, slave_tag, MPI_COMM_WORLD);

If you don't like having slop at the end, you'll have to do it in two messages:

len=strlen(str) + 1;  /* +1 for the NULL byte */
MPI_Send(&len, 1, MPI_INT, slave_rank, slave_tag, MPI_COMM_WORLD);
MPI_Send(str, strlen(str), MPI_CHAR, slave_rank, slave_tag, MPI_COMM_WORLD);

and you'd match that with

MPI_Recv(&len, 1, MPI_INT, master_rank, slave_tag, MPI_COMM_WORLD);
payload= malloc(len);
MPI_Recv(&payload, len, MPI_CHAR, master_rank, slave_tag, MPI_COMM_WORLD);

edited Sep 23 '14 at 16:27

answered Sep 23 '14 at 15:14

Rob Latham

5,085
3
27
44

The question is about an MPI implementation in Java. – Hristo Iliev Sep 23 '14 at 15:18
The idea is still the same unless your Java MPI implementation has some fancy String object. You'll just have to change `strlen(str)` to `str.length()`. – Wesley Bland Sep 23 '14 at 15:23
Hristo, thanks for pointing that out to me. I edited my answer to make it clear that I don't know anything about Java – Rob Latham Sep 23 '14 at 16:28
Thanks a lot for all your detailed answers .... Yesterday, I discovered a very easy way to do that in java ... You just need to transform your String Arrayy to an Object Array and then you send the length of the Array object : MPI.COMM_WORLD.Isend(ObjectArray, index, ObjectArray.length, MPI.OBJECT, rank, tag); – Study Learn Sep 23 '14 at 17:21
@StudyLearn, wish I could have seen your comment before I punched all that code in my answer :) – Hristo Iliev Sep 23 '14 at 17:51
Oh sorry really ... My answer below. – Study Learn Sep 23 '14 at 19:05
The question is about sending an ARRAY of strings, not just one string – Kelvin Bouma Jun 16 '23 at 14:10

Sending an Array of Strings using Message Passing Interface

2 Answers2

Sender

Receiver