Why is bash breaking MPI job control loop

Question

I'm attempting to use a simple bash script to sequentially run a batch of MPI jobs. This script works perfectly when running serial code (I am using Fortran 90), but for some reason bash breaks out of the loop when I attempt to execute MPI code.

I already found a work-around to the problem. I just wrote essentially the exact same script in Perl and it worked like a charm. I just really want to understand the issue here because I prefer the simplicity of bash and it perfectly fits my own scripting needs in almost all other cases.

I've tried running the MPI code as a background process and using wait with the same result. If I run the jobs in the background without using wait, bash does not break out of the loop, but it stacks up jobs until eventually crashing. The goal is to run the executable sequentially for each parameter set anyway, I just wanted to note that the loop is not broken in that case.

Bash Script, interp.sh: Usage --> $ ./interp.sh inputfile

#!/bin/bash

PROG=$1
IFILE=$2

kount=0 # Counter variable for looping through input file
sys=0 # Counter variable to store how many times model has been run
while IFS="\n" read -r line
do
    kount=$(( $kount + 1 ))
    if [ $(( kount % 2 )) -eq 1 ] # if kount is even, then expect headers
    then
        unset name defs
        sys=$(( $sys + 1 ))
        name=( $line ) # parse headers
        defs=${#name[*]}
        k=$(( $defs - 1 ))

    else # if count is odd, then expect numbers
        unset vals
        vals=( $line ) # parse parameters
        for i in $( seq 0 $k )
        do
            # Define variables using header names and set their values
            printf -v "${name[i]}" "${vals[i]}"
        done

        # Print input variable values
        echo $a $b $c $d $e $nPROC
        # Run executable
        mpiexec -np $nPROC --oversubscribe --hostfile my_hostfile $PROG
    fi
done < $IFILE

Input file, input.dat:

a   b   c   d   e   nPROC
1   2   3   4   5    2
nPROC
3
nPROC
4
nPROC
5
nPROC
6
nPROC
7
nPROC
8

Sample MPI f90 code, main.f90:

program main
use mpi
implicit none
integer :: i, ierr, myID, nPROC
integer, parameter :: foolen = 100000

double precision, dimension(0:foolen) :: foo

call MPI_INIT(ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, nPROC, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, myID, ierr)

if ( myID .eq. 0 ) then
    do i=0,foolen
        foo(i) = i
    end do
else
    do i=0,foolen
        foo(i) = i
    end do
end if
call MPI_FINALIZE(ierr)

end program

Sample makefile:

COMP=mpif90
EXT=f90
CFLAGs=-Wall -Wextra -Wimplicit-interface -fPIC -fmax-errors=1 -g -fcheck=all \
       -fbacktrace
MPIflags=--oversubscribe --hostfile my_hostfile

PROG=main.x
INPUT=input.dat
OUTPUT=output
OBJS=main.o

$(PROG): $(OBJS)
    $(COMP) $(CFLAGS) -o $(PROG) $(OBJS) $(LFLAGS)

main.o: main.f90
    $(COMP) -c $(CFLAGs) main.f90

%.o: %.f90
    $(COMP) -c $(CFLAGs) $<

run:
    make && make clean
    ./interp.sh $(PROG) $(INPUT)

clean:
    rm -f *.o DONE watch

my_hostfile

localhost slots=4

Note that if the mpiexec line is commented out, the script runs as expected. The output looks like this:

1 2 3 4 5 2
1 2 3 4 5 3
1 2 3 4 5 4
1 2 3 4 5 5
1 2 3 4 5 6
1 2 3 4 5 7
1 2 3 4 5 8

These are the parameter values which are supposed to be passed to the MPI code in each loop. However, when mpiexec is called in the script, only the first set of parameters is read and passed.

I apologize if all that is a bit excessive, I just wanted to provide all that is needed for testing. Any help solving the issue in bash or explanation of why this happens would be greatly appreciated!

`mpiexec` might read from `stdin`. If applicable, try `mpiexec ... < /dev/null` — Gilles Gouaillardet, May 06 '19 at 00:03
Thanks a lot @GillesGouaillardet! That solved the issue. If anyone out there is looking for more information on this, look [here](https://stackoverflow.com/questions/19955260/what-is-dev-null-in-bash). — Clint Jordan, May 06 '19 at 04:40

score 0 · Answer 1 · answered Apr 07 '22 at 08:29

mpiexec is consuming the stdin thus reading all remaining lines in the loop. So after the first loop stdin is empty and the loop breaks.

This is an issue that occurs not only with loops calling mpiexec from whithin but also with loops other commands that consumes stdin by default such as ssh.

The general solution is to use < /dev/null so that the offending command won't consume stdin but the /dev/null instead. Some commands have special flags to replace the redirect command such as ssh -n

so the solution in this case would be to add the redirect at the end of the line where mpiexec is called:

mpiexec -np $nPROC --oversubscribe --hostfile my_hostfile $PROG < /dev/null

there are some issues to pay attention to in the case of mpiexec related to Standard I/O detailed here: https://www.open-mpi.org/doc/v3.0/man1/mpiexec.1.php#toc14

Why is bash breaking MPI job control loop

1 Answers1