14

I have a batch file to send a job with sbatch. The contents of the batch file is

# Setting the proper SBATCH variables 
...
#SBATCH --error="test_slurm-%j.err"
#SBATCH --output="test_slurm-%j.out"
...

WORKDIR=.
echo "Run 1"
${WORKDIR}/test_slurm
echo "Run 2"
${WORKDIR}/test_slurm

File test_slurm-%j.out is sometimes appended with output only after each of the lines is finished. For instance, the output of the first instance of test_slurm would not be dumped into test_slurm-%j.out until test_slurm finishes.

Is there any way of having stdout/stderr "flushed" at runtime, so I can check file test_slurm-%j.out to monitor execution?

As per How to change how frequently SLURM updates the output file (stdout)? and https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe I tried with

stdbuf -oL -eL ${WORKDIR}/test_slurm
stdbuf -o0 -e0 ${WORKDIR}/test_slurm
stdbuf -o1 -e1 ${WORKDIR}/test_slurm
unbuffer ${WORKDIR}/test_slurm

None of this worked.

This does not seem to provide the answer.

sancho.s ReinstateMonicaCellio
  • 14,708
  • 20
  • 93
  • 185
  • Sorry if this is obvious but is working directory is shared among compute nodes? – damienfrancois Jun 12 '17 at 07:57
  • Probably not slurm's doing. I don't believe it messes with buffering, rather it simply does whatever the script tells it to. There isn't really enough information here to go on. Try running the batch file without slurm and observe the buffering behavior. – John Lunzer Dec 14 '18 at 13:10

0 Answers0