Why does my variable set in a do loop disappear? (unix shell)

Question

This part of my script is comparing each line of a file to find a preset string. If the string does NOT exist as a line in the file, it should append it to the end of the file.

STRING=foobar
cat "$FILE" | while read LINE
    do
        if [ "$STRING" == "$LINE" ]; then
            export ISLINEINFILE="yes"
        fi
    done
    if [ ! "$ISLINEINFILE" == yes ]; then
        echo "$LINE" >> "$FILE"
    fi

However, it appears as if both $LINE and $ISLINEINFILE are both cleared upon finishing the do loop. How can I avoid this?

Please note the shell you are working with. For this issue `ksh` seems to be better, for the future consider `bash`. Look for `shebang` how to force the correct shell for your script. — Walter A, Aug 06 '16 at 13:48
See 1) [UUOC](https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat), 2) [correct-bash-and-shell-script-variable-capitalization](http://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization), 3) [I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?](http://mywiki.wooledge.org/BashFAQ/024) and 4) [why-is-using-a-shell-loop-to-process-text-considered-bad-practice](http://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice). — Ed Morton, Aug 08 '16 at 00:55

score 1 · Answer 1 · edited Apr 13 '17 at 12:36

Why does my variable set in a do loop disappear?

It disappears because it is set in a shell pipeline component. Most shells run each part of a pipeline in a subshell. By Unix design, variables set in a subshell cannot affect their parent or any already running other shell.

How can I avoid this?

There are several ways:

The simplest is to use a shell that doesn't run the last component of a pipeline in a subshell. This is ksh default behavior, e.g. use that shebang:

#!/bin/ksh

This behavior can also be bash one when the lastpipe option is set:

shopt -s lastpipe
You might use the variable in the same subshell that set it. Note that your original script indentation is wrong and might lead to the incorrect assumption that the if block is inside the pipeline, which isn't the case. Enclosing the whole block with parentheses will rectify that and would be the minimal change (two extra characters) to make it working:

STRING=foobar
cat "$FILE" | ( while read LINE
    do
        if [ "$STRING" == "$LINE" ]; then
            export ISLINEINFILE="yes"
        fi
    done
    if [ ! "$ISLINEINFILE" == yes ]; then
        echo "$LINE" >> "$FILE"
    fi
    )

The variable would still be lost after that block though.

You might simply avoid the pipeline, which is straigthforward in your case, the cat being unnecessary:

STRING=foobar
while read LINE
do
    if [ "$STRING" == "$LINE" ]; then
        export ISLINEINFILE="yes"
    fi
done < "$FILE"
if [ ! "$ISLINEINFILE" == yes ]; then
    echo "$LINE" >> "$FILE"
fi

You might use another argorithmic approach, like using sed or gawk as suggested by John1024.

See also https://unix.stackexchange.com/a/144137/2594 for standard compliance details.

The indentation is irrelevant, and this is a useless use of `cat` that isn't worth preserving. — chepner, Aug 06 '16 at 03:34

John1024 · Accepted Answer · 2016-08-06T02:35:31.320

Using shell

If we want to make just the minimal change to your code to get it working, all we need to do is switch the input redirection:

string=foobar
while read line
do
    if [ "$string" == "$line" ]; then
        islineinfile="yes"
    fi
done <"$file"
if [ ! "$islineinfile" == yes ]; then
    echo "$string" >> "$file"
fi

In the above, we changed cat "$file" | while do ...done to while do...done<"$file". With this one change, the while loop is no longer in a subshell and, consequently, shell variables created in the loop live on after the loop completes.

Using sed

I believe that the whole of your script can be replaced with:

sed -i.bak '/^foobar$/H; ${x;s/././;x;t; s/$/\nfoobar/}' file*

The above adds line foobar to the end of each file that doesn't already have a line that matches ^foobar$.

The above shows file* as the final argument to sed. This will apply the change to all files matching the glob. You could list specific files individually if you prefer.

The above was tested on GNU sed (linux). Minor modifications may be needed for BSD/OSX sed.

Using GNU awk (gawk)

awk -i inplace -v s="foobar" '$0==s{f=1} {print} ENDFILE{if (f==0) print s; f=0}' file*

Like the sed command, this can tackle multiple files all in one command.

Why does my variable set in a do loop disappear? (unix shell)

2 Answers2

Using shell

Using sed

Using GNU awk (gawk)

Linked

Related