2

I'm using a library that generates a whole ton of output to stderr (and there is really no way to suppress the output directly in the code; it is ROOT's Minuit2 minimizer which is known for not having a way to suppress the output). I'm running batch jobs through the LSF system, and the error output files are so big that they exceed my disk quota. Erk.

When I run locally on a shell, I do:

python main.py 2> >( grep -v Minuit2 2>&1 )

to suppress the output, as is done here. This works great, but unfortunately I can't seem to get that or any variation of it to work when running on LSF. I think this is due to LSF not spawning the necessary subshell, but it's not clear.

I run on batch by passing LSF a submit script. The relevant line is:

python main.py $INPUT_FILE

which works great, aside from the aforementioned problem of gigantic error files.

When I try changing that line to

python main.py $INPUT_FILE 2> >( grep -v Minuit2 2>&1 )

I end up with

./singleSubmit.sh: line 16: syntax error near unexpected token `>'
./singleSubmit.sh: line 16: `python $MAIN $1 2> >( grep -v Minuit2 2>&1 )'

in the error log file.

Any idea how I could accomplish what I want, or why this is not working?

Thanks a ton!

Community
  • 1
  • 1
DenverCoder9
  • 473
  • 3
  • 9

1 Answers1

1

The syntax you're using works in bash, not in csh/tcsh. Try changing the first line of your submission script to

#!/bin/bash
Squirrel
  • 2,262
  • 2
  • 18
  • 29
  • Huh. I thought I tried that. Thank you so much for the help!! :) – DenverCoder9 Nov 03 '14 at 15:20
  • 1
    Out of curiosity, why does `python main.py $INPUT_FILE 2> >( grep -v Minuit2 2>&1 )` work in interactive tcsh? Shouldn't I get the same error? – DenverCoder9 Nov 03 '14 at 15:21
  • That's a good question, because I don't think it should. I'm running tcsh interactively and can't get it to work: the error message I get is "Missing name for redirect" – Squirrel Nov 03 '14 at 15:36
  • 1
    Yeah, turns out I was just being silly - I was actually running interactively in zsh. I get the same result as you for tcsh. Cheers! – DenverCoder9 Nov 03 '14 at 16:18