How does one specify physical CPU ids in openmpi 1.6.4+ rankfiles?

Question

The latest (v1.8.3) OpenMPI documentation specifies that rankfiles must now use the logical cpu IDs reported by hwloc rather than the physical IDs, see the last sentence in the Rankfiles section of the mpirun documentation here:

Starting with Open MPI v1.7, all socket/core slot locations are be specified as logical indexes (the Open MPI v1.6 series used physical indexes). You can use tools such as HWLOC’s "lstopo" to find the logical indexes of socket and cores.

I've noticed a few questions on this site (notably this question and the answer to this question) that indicate that one can specify physical cpu ids in an openMPI rankfile by prefixing the id with a p. For example:

rank 0=localhost slot=p0
rank 1=localhost slot=p8
rank 2=localhost slot=p1
rank 3=localhost slot=p9

To request physical cpu id 0 for rank 0, physical cpu id 8 for rank 1 etc...

I've tried searching for this syntax in the OpenMPI docs to no avail. I've also tried to have someone actually try constructing a rankfile this way in OpenMPI 1.6.4, which he reported also doesn't work.

What version(s) of OpenMPI does this syntax work with? Is it documented anywhere? What is the formal syntax?

Use `lstopo -l` to get the logical IDs. Then put in the rankfile `slot=logicalID`. — Hristo Iliev, Nov 13 '14 at 20:21
Thanks. I realize that you can get the logical IDs this way, and that may be fine if you're creating your rankfiles manually. However, if you're auto-generating the rankfile as part of a job execution setup then you'll need to do a call to remotely call `lstopo` for each host referenced in the rankfile. That's just not acceptable for our use case. That's why I'm curious about the versions of openmpi that support the `p` notation for specifying physical ids, and the specific syntax of that notation. — Squirrel, Nov 13 '14 at 20:35
May [the source](https://svn.open-mpi.org/trac/ompi/browser/trunk/opal/mca/hwloc/base/hwloc_base_util.c#L1189) be with you, or, in that particular case, against you. — Hristo Iliev, Nov 13 '14 at 23:50

score 1 · Accepted Answer · edited May 23 '17 at 10:33

Thanks to Hristo Iliev for pointing me in the direction of the appropriate code. It seems the function hwloc_base_slot_list_parse appeared in the open-mpi code from version 1.8.

Tracing back through the code I arrived at the orte_rmaps_rankfile_parse function which seems to go back as far as version 1.3. Looking into the history of this function, we find that the following code snippet appears from the version 1.5 branch on in the section parsing the slot list:

/* we no longer support physical mappings */
if ('P' == value[0] || 'p' == value[0]) {
    orte_show_help("help-rmaps_rank_file.txt", "not-supported", true, rankfile);
    rc = ORTE_ERR_SILENT;
    ORTE_ERROR_LOG(rc);
    goto unlock;
}

So from this I conclude that the answer to my question is that the p notation is supported in OpenMPI versions below 1.5

Edit: I also found this message in the Open MPI Users mailing list which seems to support my findings.

How does one specify physical CPU ids in openmpi 1.6.4+ rankfiles?

1 Answers1