1

Possible duplicate

Hello,

I'm struggling with output redirection within a bash for loop. I have several similar bash scripts, each scripts being launched on a different input file. The scripts called a tool with a few arguments.

The bash scripts are like this :

my_tool --input input_file --arg1 arg1_name --arg2 arg2_name > output_dir/result_X.dat

X being a string which is unique for each script.

I run all of them in a for loop :

for script in scripts_dir/*.sh
do
    bash $script
done

However, the stdout of each run still display on the terminal. The specified output files are created, but empty. How I can solve that ? I found other questions on stackoverflow where the answer is a redirection of the full loop in one big output file, but I'd like to avoid that.

If I replace > output_dir/result_X.dat by > /dev/null : the standard outputs display on the terminal

If I replace > output_dir/result_X.dat by ` 2> /dev/null : nothing is displayed.

Thanks in advance.

Micawber
  • 707
  • 1
  • 5
  • 19
  • 6
    Are you sure the program isn't writing to standard *error*? – chepner Oct 01 '19 at 14:31
  • 1
    What is `an_identifier_defined_in_the_script.dat`? The shell parses that line before the script is even executed, so if you are trying to define the name of the output file in the script, this will not work. – William Pursell Oct 01 '19 at 14:34
  • @WilliamPursell it's simply a string for ID. The script contains a command line for another tool, and the string changes for every run (but there is a different bash file for every run, so the ID strings are already written) – Micawber Oct 01 '19 at 14:38
  • @Amessihel I just tried with one file : nothing displays on the terminal, but the output file is still empty :/ – Micawber Oct 01 '19 at 14:41
  • @Micawber : so the code you are having issues with is not the code you have shown. You should be able to reproduce your issue in a minimal way. What happens if you execute `my_script_or_program > /dev/null`? Very likely the loop is irrelevant. Given that `2> /dev/null` gives you no output, it is pretty clear that all of your script's output is being written to stderr. – William Pursell Oct 01 '19 at 14:41
  • OK it's a bit confusing for me, I'' do some tests and update the question with details – Micawber Oct 01 '19 at 14:46
  • I added `2>&1` after `> output_dir/result_X.dat` and it seems to work fine now. Can anyone explain shy ? – Micawber Oct 01 '19 at 15:20
  • 1
    Yes, it is because your program is outputting to standard error! I think we said this before? `2>&1`says redirect stderr to stdout. You could equally put `2>output/result_X.dat` – Gem Taylor Oct 01 '19 at 16:19

1 Answers1

2

When you start my_tool, there are normally 3 file-descriptors available in the tool:

  • STDIN
  • STDOUT
  • STDERR

STDIN is used for input, and therefore irrelevant for this question. STDOUT is used for standard output. This is file-descriptor 1. If you do 

ls 1> /dev/null

the STDOUT of ls is written to /dev/null. If you do not add the 1, as in ls > /dev/null, it is assumed that you mean STDOUT.

STDERR is used as standard output for error messages, in the broadest sense of the word. The number of STDERR is 2.

Using ls instead of your my_command, ls > file will put the listing in the file. ls /non_existing_dir_ > file will put the STDOUT of ls in the file. But there is no output on STDOUT, and because STDERR is not redirected, it will be send to the terminal.

So, to conclude,

ls . /non_existing_dir 2>stderr >stdout

will put the directory listing of . in the file stdout and the error for the non-existing directory in stderr.

With 2>&1 you redirect the output of filedescriptor2 (STDERR) to file descriptor 1 (SDTOUT).

To complicate things a bit, you can add other file descriptor numbers. For example:

exec 3>file

will put the output of file descriptor 3 (which is newly created) in file. And 

ls 1>&3

will then redirect the output of file descriptor 1 to file descriptor 3, effectively putting the output of ls in file.

Ljm Dullaart
  • 4,273
  • 2
  • 14
  • 31
  • I know part of that already, but I had not clue this program was sending the output to stderr ! Silly me. – Micawber Oct 01 '19 at 19:56