124

Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:

#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output

if [ -z "$OUTPUT" ]
then
        echo "Status WARN: No messages from SMcli"
        exit $STATE_WARNING
else
        echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
        do
                if [ "$STATUS" != "Optimal" ]
                then
                        echo "CRIT: $NAME - $STATUS"
                        echo $((++XCODE))
                else
                        echo "OK: $NAME - $STATUS"
                fi
        done
fi

echo $XCODE

I've tried using the following statement instead of the ++XCODE method

XCODE=`expr $XCODE + 1`

and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.

Matt P
  • 5,447
  • 4
  • 23
  • 20

7 Answers7

157

Because you're piping into the while loop, a sub-shell is created to run the while loop.

Now this child process has its own copy of the environment and can't pass any variables back to its parent (as in any unix process).

Therefore you'll need to restructure so that you're not piping into the loop. Alternatively you could run in a function, for example, and echo the value you want returned from the sub-process.

http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL

Community
  • 1
  • 1
pixelbeat
  • 30,615
  • 9
  • 51
  • 60
  • 12
    this just answered so many of the seemingly random issues i was running into with bash scripting. – Daniel Agans Jan 27 '15 at 14:29
  • 1
    This perfect answer upsets me so much and explains a really weird behaviour in our CI system. – KayCee Apr 06 '17 at 08:29
  • This _great_ answer would be improved with an example for the OP: `while read NAME IP1 IP2 STATUS; do stuff-in-body-of-while-loop; done < <(echo "name1 ip ip status")`, replacing the `echo` statement in that [process substitution](https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html) with the "[other] command with multi line output" mentioned in the original question. Pro Tip: [ShellCheck](https://www.shellcheck.net/) will [advise you](https://www.shellcheck.net/wiki/SC2162) to use `read -r` to avoid corrupting the input if it happens to contain backslashes. – TheDudeAbides Oct 21 '22 at 17:00
125

The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.

Your specific example can be solved by rewriting the pipe to

while ... do ... done <<< "$OUTPUT"

or perhaps

while ... do ... done < <(echo "$OUTPUT")
mweerden
  • 13,619
  • 5
  • 32
  • 32
  • 34
    For those who are looking on at this confused as to what the whole <() syntax is (like I was), it's called "Process Substitution", and the specific usage detailed above can be seen here: http://mywiki.wooledge.org/ProcessSubstitution – Ross Aiken Feb 19 '13 at 14:53
  • 4
    Process Substitution is something everyone should be using regularly! It is super useful. I do something like `vimdiff <(grep WARN log.1 | sort | uniq) <(grep WARN log.2 | sort | uniq)` every day. Consider that you can use multiple at once and treat them like files... POSSIBILITIES! – Bruno Bronosky Nov 08 '18 at 19:29
  • 1
    I had this exact issue, and refactoring the pipline to save the first stage output and write like this was trivially simple. Thank you so much for saving me hours of research and experimentation. – jwm Jun 12 '21 at 00:22
13

This should work as well (because echo and while are in same subshell):

#!/bin/bash
cat /tmp/randomFile | (while read line
do
    LINE="$LINE $line"
done && echo $LINE )
sano
  • 131
  • 1
  • 2
3

One more option:

#!/bin/bash
cat /some/file | while read line
do
  var="abc"
  echo $var | xsel -i -p  # redirect stdin to the X primary selection
done
var=$(xsel -o -p)  # redirect back to stdout
echo $var

EDIT: Here, xsel is a requirement (install it). Alternatively, you can use xclip: xclip -i -selection clipboard instead of xsel -i -p

Rammix
  • 31
  • 1
  • 6
  • I get an error: ./scraper.sh: line 111: xsel: command not found ./scraper.sh: line 114: xsel: command not found – 3kstc Apr 19 '16 at 01:57
  • @3kstc obviously, install xsel. Also, you can use xclip, but its usage a little bit different. Main point here: 1st you put the output into a clipboard (3 of them in linux), 2nd - you grab it from there and send to stdout. – Rammix May 09 '16 at 20:43
3

I got around this when I was making my own little du:

ls -l | sed '/total/d ; s/  */\t/g' | cut -f 5 | 
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )

The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.

Adrian May
  • 2,127
  • 15
  • 24
3

Another option is to output the results into a file from the subshell and then read it in the parent shell. something like

#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
    LINE="$LINE $line"
    echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)
David Newcomb
  • 10,639
  • 3
  • 49
  • 62
freethinker
  • 1,286
  • 1
  • 10
  • 17
2
 #!/bin/bash
 OUTPUT="name1 ip ip status"
+export XCODE=0;
 if [ -z "$OUTPUT" ]
----

                     echo "CRIT: $NAME - $STATUS"
-                    echo $((++XCODE))
+                    export XCODE=$(( $XCODE + 1 ))
             else

echo $XCODE

see if those changes help

Kent Fredric
  • 56,416
  • 14
  • 107
  • 150
  • When doing this, I now get a "0" to print for the last echo statement. however I expect the value to be 1 not zero. Also, why the use of export? I assume that forces it into the environment? – Matt P Sep 23 '08 at 22:09