\n in variable in heredoc

Question

Is there any way to for a Bash heredoc to interpret '\n\' in a heredoc?

I have an iteratively built string in a loop, something like

for i in word1 word2 word3
do
        TMP_VAR=$i
        ret="$ret\n$TMP_VAR"
done

and then I want to use the created string in a heredoc:

cat <<EOF > myfile
HEADER
==
$ret
==
TRAILER
EOF

however I would like to interpret the "\n" character as newline, so that the output is

HEADER
==
word1
word2
word3
==
TRAILER

instead of

HEADER
==
\nword1\nword2\nword3
==
TRAILER

Is it possible? Or should I perhaps build my initial string somehow otherwise?

Answer 1

In bash you can use $'\n' to add a newline to a string:

ret="$ret"$'\n'"$TMP_VAR"

You can also use += to append to a string:

ret+=$'\n'"$TMP_VAR"

Answer 2

As others (and other answers to other questions) have said, you can put encoded characters into a string for the shell to interpret.

x=$'\n' # newline
printf -v x '\n' # newline

That said, I don't believe there is any way to directly put an encoded newline into a heredoc.

cat <<EOF
\n
EOF

just outputs a literal \n

cat <<$'EOF'
…
EOF

is nothing special, nor is <<'EOF'

The best you can do is to preencode the newline, and include the expansion in the heredoc:

nl=$'\n'
cat <<EOF
foo bar $nl baz
EOF

outputs

foo bar
 baz

Answer 3

Change:

==
$ret
==

to:

==
$(echo -e $ret)
==

Answer 4

The best solution is to build your variable with actual newlines, instead of inserting character sequences which need to be replaced with newlines.

I find the following function sufficiently useful that I put it in my bash startup file; for you simple case, it would work perfectly:

lines() { printf %s\\n "$@"; }

With that, you could write:

ret=$(lines word1 word2 word3)

instead of the loop you use. Then you can insert $ret into the heredoc and it will work as expected. [See note 1]

However, if for whatever reason you really want to construct your string with escape sequences instead of actual characters, you can do the expansion using an extended feature in the bash printf built-in, the %b format code. %b does almost the same escape conversions, but there are a couple of differences. See help printf for details. Using that you could do the following:

$ ret="word1\nword2\nword3"
$ cat <<EOF > tmp
> HEADER
> ==
> $(printf "%b" "$ret")
> ==
> TRAILER
> EOF
$ cat tmp
HEADER
==
word1
word2
word3
==
TRAILER

Notes

1. There is a subtlety in the use of the lines function. printf keeps repeating its format string until it absorbs all of its arguments, so that the format %s\\n puts a newline after every argument, including the last one. For most use cases, that's exactly what you want; most of my uses of lines have to do with feeding the result into a utility which expects lines of inputs.

   But in the case of ret=$(lines word1 word2 word3), I didn't really want the trailing newline, since my plan is to insert $ret on a line by itself in the here doc. Fortunately, command substitution ($(...)) always deletes trailing newlines from the output of the command, so the value of ret after the assignment has newlines between the arguments, but not at the end. (This feature is occasionally annoying but more often it is exactly what you wanted, so it goes unnoticed.)

Answer 5

Using awk '{print}' solves the problem for me.

cat << EOF |
line1
line2
line3
EOF
awk '{print}' > outputfile
# outputfile contents
cat outputfile
line1
line2
line3