Use gdb to debug MPI in multiple screen windows

Question

If I have an MPI program that I want to debug with gdb while being able to see all of the separate processes' outputs, I can use:

mpirun -n <NP> xterm -hold -e gdb -ex run --args ./program [arg1] [arg2] [...]

which is well and good when I have a GUI to play with. But that is not always the case.

Is there a similar set up I can use with screen such that each process gets its own window? This would be useful for debugging in a remote environment since it would allow me to flip between outputs using Ctrl+a n.

score 2 · Answer 1 · edited May 23 '17 at 12:14

I think this answer in the "How do I debug an MPI program?" thread does what you want.

EDITS:

In response to the comment, you can do it somewhat more easily, although succinct isnt exactly the term I would use:

Launch a detached screen via mpirun - running your debugger and process. I've called the session mpi, and im passing through my library path because it gets stripped by screen and my demo needs it (also I'm on a mac, hence lldb and DYLD):

mpirun -np 4 screen -AdmS mpi env DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH lldb demo.out

Then launch a seperate screen session, which i've called 'debug':

screen -AdmS debug

Use screen -ls to list the running sessions:

>> screen -ls

There are screens on:

19871.mpi (Detached)

19872.mpi (Detached)

19875.mpi (Detached)

19876.mpi (Detached)

20105.debug (Detached)

Now launch 4 new tabs in the debug session, attaching each to one of the mpi sessions:

screen -S debug -X screen -t tab0 screen -r 19871.mpi
screen -S debug -X screen -t tab1 screen -r 19872.mpi
screen -S debug -X screen -t tab2 screen -r 19875.mpi
screen -S debug -X screen -t tab3 screen -r 19876.mpi

Then simply attach to your debug session with screen -r debug. Now you have 4 tabs, each running a serial instance of the debugger attached to an mpi process similarly to the xterm method you described before. Its not exactly the quickest set of commands, but at least you dont need to modify your code or chase PIDs etc.

Another method I tried, but doesnt seem to work:

Launch a detached screen

screen -AdmS ashell

Launch two mpi processes that start new screen tabs in the detached session, launching lldb with my demo mpi application:

mpirun -np 1 screen -S ashell -X screen -t tab1 env DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH lldb demo.out : -np 1 screen -S ashell -X screen -t tab2 env DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH lldb demo.out

Or alternatively just

mpirun -np 2 screen -S ashell -X screen env DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH lldb demo.out

Then attach to screen with

screen -r ashell

And you'll have 3 tabs, 2 of them running lldb with your program, and one with whatever your standard shell is. Unfortunately when you try running the programs, each process thinks its the only one in the comm world, and im not sure what to do about that...

It seems so convoluted compared to the way you can start xterm. Is there not some more succinct method? — Richard, Sep 18 '15 at 16:03
Im not sure - I edited the answer with how i would have expected to do it, but it doesnt work. It might get you started though. — timdykes, Sep 19 '15 at 20:33
@Richard - Ive added a more detailed answer with the 'simplest' way I could figure out to do what you want. Its a bit easier than the method in the linked post, although still not quite as easy as the xterm method. — timdykes, Sep 19 '15 at 21:24
Thanks, @timofiend. I was able to modify your answer (see [here](http://stackoverflow.com/a/32685186/752843)) using xargs to automatically generate the `debug` screen session. This allowed me to build a handy script to run everything. — Richard, Sep 20 '15 at 23:06
Actually, neither of our cod seems to be catching the arguments to the command. And, when I use `-ex run --args `, the program never starts, as though each process is waiting to, and failing, to communicate. — Richard, Sep 21 '15 at 00:01
Hmm I dont usually pass anything, I start from the lldb interactive menu. One workaround is to use a script to send commands to every window similar to what the guy did in the post I linked, then use it to launch your program from gdb interactively — timdykes, Sep 21 '15 at 15:58

score 2 · Answer 2 · edited May 23 '17 at 12:06

How do you debug a C/C++ MPI program?

One way is to start a separate terminal and gdb session for each of the processes:

mpirun -n <NP> xterm -hold -e gdb -ex run --args ./program [arg1] [arg2] [...]

where NP is the number of processes.

What if you don't have a GUI handy?

(See below for a handy script.)

This is based on timofiend's answer here.

Spin up the mpi program in its debugger in a number of screen sessions:

mpirun -np 4 screen -AdmS mpi gdb ./parallel_pit_fill.exe one retain ./beauford.tif 500 500

Spin up a new screen session to access the debugger:

screen -AdmS debug

Load the debugger's screen sessions in to the new screen session

screen -list |                   #Get list of screen sessions
   grep -E "[0-9]+.mpi"  |       #Extract the relevant ones
   awk '{print NR-1,$1}' |       #Generate tab #s and session ids, drop rest of the string
   xargs -n 2 sh -c '
     screen -S debug -X screen -t tab$0 screen -r $1
   '

Jump into the new screen session:

screen -r debug

I've encapsulated the above in a handy script:

#!/bin/bash
if [ $# -lt 2 ]
then
  echo "Parallel Debugger Syntax: $0 <NP> <PROGRAM> [arg1] [arg2] [...]"
  exit 1
fi

the_time=`date +%s` #Use this so we can run multiple debugging sessions at once
                    #(assumes we are only starting one per second)

#The first argument is the number of processes. Everything else is what we want
#to run. Make a new mpi screen for each process.
mpirun -np $1 screen -AdmS ${the_time}.mpi gdb "${@:2}"

#Create a new screen for debugging from
screen -AdmS ${the_time}.debug

#The following are used for loading the debuggers into the debugging screen
firstpart="screen -S ${the_time}.debug"
secondpart=' -X screen -t tab$0 screen -r $1'

screen -list |                         #Get list of mpi screens
   grep -E "[0-9]+.${the_time}.mpi"  | #Extract the relevant ones
   awk '{print NR-1,$1}' |             #Generate tab #s and session ids, drop rest of the string
   xargs -n 2 sh -c "$firstpart$secondpart"

screen -r ${the_time}.debug            #Enter debugging screen

Warning: this doesn't seem to pick up arguments to the program! Using `-ex run --args ` results in the program hanging at `MPI_INIT` as though it cannot contact the other processes. I am not sure why this is. — Richard, Sep 21 '15 at 00:20

philb · Answer 3 · 2021-10-28T20:01:39.113

1

You can have a look at tmpi, which automates what the other answers show how to achieve, but using tmux instead of screen.

And as a bonus, it multiplexes your keyboard input to all MPI ranks!

edited Oct 28 '21 at 20:01

answered Oct 28 '21 at 16:50

philb

2,031
15
20

Use gdb to debug MPI in multiple screen windows

3 Answers3

Linked