How to determine MPI rank/process number local to a socket/node

Question

Say, I run a parallel program using MPI. Execution command

mpirun -n 8 -npernode 2 <prg>

launches 8 processes in total. That is 2 processes per node and 4 nodes in total. (OpenMPI 1.5). Where a node comprises 1 CPU (dual core) and network interconnect between nodes is InfiniBand.

Now, the rank number (or process number) can be determined with

int myrank;
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

This returns a number between 0 and 7.

But, How can I determine the node number (in this case a number between 0 and 3) and the process number within a node (number between 0 and 1)?

score 20 · Answer 1 · answered Oct 19 '16 at 05:15

20

I believe you can achieve that with MPI-3 in this manner:

MPI_Comm shmcomm;
MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0,
                    MPI_INFO_NULL, &shmcomm);
int shmrank;
MPI_Comm_rank(shmcomm, &shmrank);

answered Oct 19 '16 at 05:15

Dave O.

301
2
2

1

@aleixrocks That link describing MPI_Comm_split_type is failing. Here's an [updated one](https://cvw.cac.cornell.edu/mpiadvtopics/splitting), which just strips off ?Aspx... – Mark Gates Dec 17 '21 at 16:59

score 11 · Accepted Answer · edited Mar 19 '19 at 22:40

11

It depends on the MPI implementation - and there is no standard for this particular problem.

Open MPI has some environment variables that can help. OMPI_COMM_WORLD_LOCAL_RANK will give you the local rank within a node - ie. this is the process number which you are looking for. A call to getenv will therefore answer your problem - but this is not portable to other MPI implementations.

See this for the (short) list of variables in OpenMPI.

I don't know of a corresponding "node number".

edited Mar 19 '19 at 22:40

Richard

56,349
34
180
251

answered Jan 29 '12 at 10:24

David

756
5
10

Thanks for the answer. The pasted link is broken. I found this link that works https://www.open-mpi.org/faq/?category=running#mpi-environmental-variables – cxwangyi Apr 27 '21 at 19:11
2

This should not be the accepted answer. Using `MPI_Comm_split_type` you can analyze the distribution over nodes without a problem. Maybe that didn't exist when this answer was written, but right now this answer is insufficient. – Victor Eijkhout Oct 22 '21 at 18:43

score 3 · Answer 3 · answered Aug 03 '15 at 16:40

This exact problem is discussed on Markus Wittmann's Blog, MPI Node-Local Rank determination.

There, three strategies are suggested:

A naive, portable solution employs MPI_Get_processor_name or gethostname to create an unique identifier for the node and performs an MPI_Alltoall on it. [...]
[Method 2] relies on MPI_Comm_split, which provides an easy way to split a communicator into subgroups (sub-communicators). [...]
Shared memory can be utilized, if available. [...]

For some working code (presumably LGPL licensed?), Wittmann links to MpiNodeRank.cpp from the APSM library.

Strategy 2 is elegant and portable. – Victor Eijkhout Oct 22 '21 at 18:45 — Victor Eijkhout, Oct 22 '21 at 18:45

score 1 · Answer 4 · answered May 25 '18 at 09:12

Alternatively you can use

int MPI_Get_processor_name( char *name, int *resultlen )

to retreive node name, then use it as color in

int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm)

This is not as simple as MPI_Comm_split_type, however it offers a bit more freedom to split your comunicator the way you want.

How to determine MPI rank/process number local to a socket/node

4 Answers4