1

I'm looking to understand what is going on in a simple script, that seems to produce random results.

What I am trying to do:

  • Replace variables in pre-existing files from the values defined in the environment.

  • This is done inside a Docker container with a bash script, that runs the command:

    envsubst '$VAR1 $VAR2' < $FILE | tee $FILE

What happens:

  • Sometimes a $FILE in question has contents before the command, but contains nothing after the command.

How to reproduce the issue:

  • Dockerfile:
FROM debian:stretch

RUN apt-get update -qy
RUN apt-get install -qy gettext

COPY main-script /main-script
RUN chmod +x /main-script

ENTRYPOINT [ "/main-script" ]
  • Bash script:
#!/bin/bash

mkdir -p /test

export TEST1=1
export TEST2=2
export TEST3=3

for I in {1..300} ; do
    echo '$TEST1 $TEST2 $TEST3' > /test/file-$I
done

for FILE in /test/file-* ; do
    envsubst < $FILE | tee $FILE
done

for FILE in /test/file-* ; do
    if [[ -z "$(cat $FILE)" ]]; then
        echo "$FILE is empty!"
        FAIL=1
    fi
done

if [[ -n "$FAIL" ]]; then
    exit 2
fi

Output looks something like this:

...
/test/file-11 is empty!
/test/file-180 is empty!
/test/file-183 is empty!
/test/file-295 is empty!
Community
  • 1
  • 1
  • You don't need to read the file into memory to test if it is empty. `if [[ ! -s "$FILE" ]]; then` – chepner Mar 05 '20 at 13:40

1 Answers1

4

Pipes are asynchronous, and you've introduced a race condition. You can't predict if envsubst reads from $FILE before or after tee truncates it.

The correct approach is to write the changes to a temporary file, then replace the original with the temporary file after that has succeeded.

tmp=$(mktemp)
envsubst < "$FILE" > "$tmp" &&  mv "$tmp" "$FILE"
chepner
  • 497,756
  • 71
  • 530
  • 681
  • 1
    One gotcha is if `$FILE` is a symlink that points to some file. – Jetchisel Mar 05 '20 at 13:45
  • Good point. You could replace the atomic (assuming no file system boundaries are being crossed) `mv` command with something like `cat "$tmp" > "$FILE"`. – chepner Mar 05 '20 at 13:57
  • 1
    There's also the `sponge` command, which lets you write `envsubst < "$FILE" | sponge "$FILE"` because `sponge` doesn't open the file for writing until after it has read all its input. (I usually don't mention it because 1) it's non-standard and 2) I just don't have any practical experience using it.) – chepner Mar 05 '20 at 14:04