Linux cat inline file with special characters

Question

I have the following script:

#!/bin/bash
cat << EOF
A^@B^AC^B
EOF

Where:

'^@' = decimal value 0
'^A' = decimal value 1
'^B' = decimal value 2

So, the file looks like this:

xxd main.sh
00000000: 2321 2f62 696e 2f62 6173 680a 0a63 6174  #!/bin/bash..cat
00000010: 203c 3c20 454f 460a 4100 4201 4302 0a45   << EOF.A.B.C..E
00000020: 4f46 0a                                  OF.

When I run the script, decimal values 0 and 1 seem to disappear:

./main.sh | xxd
00000000: 4142 4302 0a                             ABC..

My questions is why? I was expecting to have an output like this:

Character 'A', followed by decimal value 0, followed by 'B', followed by decimal value 1, followed by 'C', followed by decimal value 2.

score 3 · Answer 1 · answered Nov 06 '17 at 15:52

The reason seems two-fold.

First, note you must quote the word/delimiter in the here-document to avoid expansion(s):

If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.

So, if you write it like this:

#!/bin/bash
cat <<"EOF"
A^@B^AC^B
EOF

you'll preserve all characters except NULL (\0).

But you can not use NULL char in either variable, or an argument of the command line (see this answer).

You can use NULLs in pipes/files though, so you might want to encode the content of your here-document and decode it on fly (for example with xxd).

Thanks, I guess I'll have to use `echo -e` instead of `cat` to include null characters. — J B, Nov 06 '17 at 17:46
You can also (probably even better) use `printf` (eg. `printf '\0\01\xCA\xFE' | hd`). If you use `echo`, be sure to also add `-n`. — randomir, Nov 06 '17 at 21:03

score 0 · Accepted Answer · answered May 07 '18 at 14:58

After thinking more about this issue, I realized that quoting the word/delimiter (as suggested by the other answer) should not affect the output of the script (parameter expansion, command substitution, and arithmetic expansion are not being performed in my script).

So, \000 indeed disappears since you cannot use NULL character in either variable, or an argument of the command line.

But, \001 disappears due to a bash bug. That value is used internally by bash as an escape character. The bug occurs in function expand_word_internal which does not properly handle that escape character (see bug report here).

The bug is already fixed in the bash devel branch.

Linux cat inline file with special characters

2 Answers2