PBS script -o file to multiple locations

Question

Sometimes when I run jobs on a PBS cluster, I'd really like the joblog (-o file) in two places. One in the $PBS_O_WORKDIR for keeping everthing together and one ${HOME}/jobOuts/ for greping/awking/etc...

Doing a test from the command line works with tee:

echo "hello" | qsub -o `tee $HOME/out1.o $HOME/out2.o $HOME/out3.o`

But once I try to put this in my PBS script, it does not work if I put it in a PBS script and qsub

####Parameterized PBS Script ####
#PBS -S /bin/bash
#PBS -l nodes=1
#PBS -l walltime=0:01:00
#PBS -j oe
#PBS -o `tee TEE_TEST.o TEE_TEST.${PBS_JOBID}.o`
#PBS -M me@email.com
#PBS -m abe
#PBS -V

cd $PBS_O_WORKDIR

echo `date`

Here is the qsub and error:

qsub  TEST.pbs
qsub: directive error: -o `tee TEE_TEST.o TEE_TEST.${PBS_JOBID}.o`

I tried a few other things below - nothing worked.

One -o line (comma, semi colon and space):

#PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o,${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o
#PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o,${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o
#PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o ${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o

and two lines:

#PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o
#PBS -o ${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o

The two works liner just takes the 2nd -o option, and the one liners don't work.

Any suggestions? Is it possible?

Answer 1

I'm very surprised that your command line example worked. Did your job actually run? My guess is that running

echo "hello" | qsub -o `tee $HOME/out1.o $HOME/out2.o $HOME/out3.o`

short circuits the job running and actually just grabs "hello" and passes it to tee. My guess is that someone who knows bash a bit better could explain what is actually happening here.

The only way I can think of to make what you're requesting work would be to have your script write things to tee. In order for this to work, you'd need to have the locations be on a network filesystem that is accessible from any possible compute node.

Answer 2

I studied a qsub man page and I don't think there is a method to specify more than one output file (each) for standard output and standard error. Taking cues from this page, here is a way I achieved something similar to your goals. Your PBS environment may be a little different. Also, I am not a bash expert, so there may be more concise methods of achieving the same thing.

Assuming you're using the default -o settings, at the end of your regular job script put the commands:

# change to the directory from which this job was submitted
cd $PBS_O_WORKDIR

# the standard output by default will be in file JOBNAME.oJOBID
# on my system, PBS_JOBID has ".machinename" at the end, which needs to be removed

filename=${PBS_JOBNAME}.o${PBS_JOBID%%.*}

echo "${PBS_O_WORKDIR}/copy.pbs $filename" | qsub

This will launch qsub using the stdin arguments telling it to run "copy.pbs" with the filename argument to be copied. The copy.pbs file I used is:

#!/bin/bash

# change to the directory from which this job was submitted
cd $PBS_O_WORKDIR

newfile=${HOME}/jobOuts/$1

cp $1 $newfile

This worked for me to copy the first PBS standard output to another directory. A side effect is that running copy.pbs with qsub creates another two output files, STDIN.e* and STDIN.o*. I figured using qsub again was a good idea to make sure the first job had finished. To be more safe, you could use the "depends on option" with qsub, such as "-W depend=afterok:$PBS_JOBID copy.pbs $filename" | qsub. But I did not test this method and like I said, I'm not an expert in any of this.