OpenMPI backtrace with line numbers

Question

OpenMPI has the MCA parameter mpi_abort_print_stack for trace backs after an MPI_ABORT, but the back trace is quite limited in details. For example, if I compile (mpicxx broadcast.cxx) and run (mpiexec --mca mpi_abort_print_stack 1 -n 4 ./a.out) this simple example code:

#include <mpi.h>

const int N = 10;
int arr[N];

int main(int argc, char** argv)
{
    int rank;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    // Correct:
    MPI_Bcast(arr, N, MPI_INT, 0, MPI_COMM_WORLD);

    // Incorrect:
    const int my_size = (rank == 1) ? N+1 : N;
    MPI_Bcast(arr, my_size, MPI_INT, 0, MPI_COMM_WORLD);

    MPI_Finalize();
    return 0;
}

I get the following output and back trace:

[manjaro:2223406] *** An error occurred in MPI_Bcast
[manjaro:2223406] *** reported by process [1082195969,3]
[manjaro:2223406] *** on communicator MPI_COMM_WORLD
[manjaro:2223406] *** MPI_ERR_TRUNCATE: message truncated
[manjaro:2223406] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[manjaro:2223406] ***    and potentially your MPI job)
[manjaro:2223406] [0] func:/usr/lib/libopen-pal.so.40(opal_backtrace_buffer+0x3b) [0x7f3681231a9b]
[manjaro:2223406] [1] func:/usr/lib/libmpi.so.40(ompi_mpi_abort+0x160) [0x7f368183f040]
[manjaro:2223406] [2] func:/usr/lib/libmpi.so.40(ompi_mpi_errors_are_fatal_comm_handler+0xb9) [0x7f36818369b9]
[manjaro:2223406] [3] func:/usr/lib/libmpi.so.40(ompi_errhandler_invoke+0xd3) [0x7f3681836ab3]
[manjaro:2223406] [4] func:/usr/lib/libmpi.so.40(PMPI_Bcast+0x455) [0x7f36818514b5]
[manjaro:2223406] [5] func:./a.out(+0x7c48) [0x560d4d420c48]
[manjaro:2223406] [6] func:/usr/lib/libc.so.6(+0x29290) [0x7f36812d2290]
[manjaro:2223406] [7] func:/usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7f36812d234a]
[manjaro:2223406] [8] func:./a.out(+0x7ac5) [0x560d4d420ac5]

So it tells me that there is a problem in some MPI_Bcast call, but not exactly which one.

Is it possible to get a more detailed back trace, including e.g. line numbers?

MPI_Bcast returns an error code, can’t you just write a handler macro for that and make sure MPI_ERRORS_RETURN is set so that the program won’t abort? Then you can report error and line number — talonmies, Feb 11 '23 at 09:12
https://www.i12.rwth-aachen.de/go/id/nrbe can help you check the correctness of a MPI application. Otherwise, parallel debuggers (such as Arm/Allinea DDT) are your best friend. Assuming you compiled with `-g`, you might be able to use `addr2line` to translate the address (0x560d4d420c48) into a file and line number. — Gilles Gouaillardet, Feb 11 '23 at 09:13
@talonmies, thanks, that's a good suggestion. I should have thought of that myself, since we use the same approach in CUDA all the time. @Gilles, thanks for pointing me in the direction of MUST. That looks like a useful tool, it also reported that there was a message size issue in my code above, but I couldn't get it to report a corresponding line number (compiled with `-O0` and `-g`). — Bart, Feb 15 '23 at 10:39

score 1 · Answer 1 · answered Feb 15 '23 at 10:34

One possibility, as suggested by @talonmies in the comments, is to define a macro which does the error checking after each MPI call. This is similar to the kind of error checking that you often see in CUDA code: What is the canonical way to check for errors using the CUDA runtime API?

#include <mpi.h>

#define check_mpi_error(n) __check_mpi_error(__FILE__, __LINE__, n)

inline void __check_mpi_error(const char *file, const int line, const int n)
{
    char errbuffer[MPI_MAX_ERROR_STRING];
    int errlen;

    if (n != MPI_SUCCESS)
    {
        MPI_Error_string(n, errbuffer, &errlen);
        printf("MPI-error: %s\n", errbuffer);
        printf("Location: %s:%i\n", file, line);
        MPI_Abort(MPI_COMM_WORLD, n);
    }
}

const int N = 10;
int arr[N];

int main(int argc, char** argv)
{
    int rank;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN);

    // Correct:
    int err;
    err = MPI_Bcast(arr, N, MPI_INT, 0, MPI_COMM_WORLD);
    check_mpi_error(err);

    // Incorrect:
    const int my_size = (rank == 1) ? N+1 : N;
    err = MPI_Bcast(arr, my_size, MPI_INT, 0, MPI_COMM_WORLD);
    check_mpi_error(err);

    MPI_Finalize();

    return 0;
}

If I compile (mpicxx -o bcast broadcast_debug.cxx) and run (mpiexec -n 4 bcast) this code, I get the following output:

MPI-error: MPI_ERR_TRUNCATE: message truncated
Location: broadcast_debug.cxx:38

OpenMPI backtrace with line numbers

1 Answers1