Bash echo command with binary data?

Question

Would someone please explain why this script sometime return only 15 bytes in hex string representation?

for i in {1..10}; do 
  API_IV=`openssl rand 16`; API_IV_HEX=`echo -n "$API_IV" | od -vt x1 -w16 | awk '{$1="";print}'`; echo $API_IV_HEX; 
done

like this:

c2 2a 09 0f 9a cd 64 02 28 06 43 f8 13 80 a5 04
fa c4 ac b1 95 23 7c 36 95 2d 5e 0e bf 05 fe f4
38 55 d3 b4 32 bb 61 f4 fd 17 92 67 e2 9b b4 04
6d a7 f8 46 e9 99 bd 89 87 f9 7f 2b 15 5a 17 8a
11 c8 89 f4 8f 66 93 f1 6d b9 2b 64 7e 01 61 68
93 e3 9d 28 95 e1 c8 92 e5 62 d9 bf 20 b3 1c dd
37 64 ef b0 2f da c7 60 1c c8 20 b8 28 9d f9
29 f0 5a e9 cc 36 66 de 02 82 fc 8e 36 bf 5d d1
b2 57 d8 79 21 df 73 1c af 07 e9 80 0a 67 c6 15
ba 77 cb 92 39 42 39 f9 a4 57 c8 c4 be 62 19 54

If pipe the "openssl rand 16" directly to the od command then it works fine, but I need the binary value. Thanks for your help.

So is there a solution for what I'm trying to accomplish which is to have both the binary and the hex string? I could output into a file and read back in, but is it the only way? — Luan Nguyen, Jul 16 '14 at 16:38

score 4 · Accepted Answer · edited May 23 '17 at 12:13

echo, like various other standard commands, considers \x00 as an end-of-string marker. So stop displaying after it.

Maybe you are looking to the -hex option of openssl rand:

sh$ openssl rand 16 -hex
4248bf230fc9dd927ab53f799e2a9708

Given that option is available on your system, your example could be rewritten:

sh$ openssl version
OpenSSL 1.0.1e 11 Feb 2013

sh$ for i in {1..10}; do 
    openssl rand 16 -hex | sed -e 's|..|& |g' -e 's| $||'
done
20 cb 6b 7a 85 2d 0b fe 9e c7 d0 4b 91 88 1b bb
5d 74 99 5e 05 c9 7d 9d 37 dd 02 f3 23 bb c5 b7
51 e9 0f dc 58 04 5e 30 e3 6b 9f 63 aa fc 95 05
fc 6b b8 cb 05 82 53 85 78 0e 59 13 3b e7 c1 4b
cf fa fc d9 1a 25 df e0 f8 59 71 a6 2c 64 c5 87
93 1a 29 b4 5a 52 77 bb 3f bb 1d 0a 46 5d c8 b4
0c bb c2 b2 b4 89 d4 37 1c 86 0a 7a 58 b8 64 e2
ee fc a7 ec 6c f8 7f 51 04 43 d6 00 d8 79 65 43
b9 73 9e cc 4b 42 9e 64 9d 5b 21 6a 20 b7 c3 16
06 8a 15 22 6a d5 ae ab 9a d2 9f 60 f1 a9 26 bd

If you need to later convert from hex to bytes use this perl one-liner:

echo MY_HEX_STRING |
  perl -ne 's/([0-9a-f]{2})/print chr hex $1/gie' |
  my_tool_reading_binary_input_from_stdin

(from https://stackoverflow.com/a/1604770/2363712)

Please note I pipe to the client program and o not use a shell variable here, as I think it cannot properly handle the \x00.

As the bash cannot properly deal with binary strings containing \x00 your best bet if you absolutely want to stick with shell programming is to use an intermediate file to store binary data. And not a variable:

This is the idea. Feel free to adapt to your needs:

for i in {1..10}; do
    openssl rand 16 > api_iv_$i # better use `mktemp` here
    API_IV_HEX=`od -vt x1 -w16 < api_iv_$i | awk '{$1="";print}'`
    echo $API_IV_HEX; 
done

sh$ bash t.sh 
cf 06 ab ab 86 fd ef 22 1a 2c bd 7f 8c 45 27 e5
2a 01 9c 7a fa 15 d3 ea 40 89 8b 26 d5 4f 97 08
55 2e c9 d3 cd 0d 3a 6f 1b a0 fe 38 6d 0e 20 07
fe 60 35 62 17 80 f2 db 64 7a af da 81 ff f7 e0
74 9a 5c 39 0e 1a 6b 89 a3 21 65 01 a3 de c4 1c
c3 11 45 e3 e0 dc 66 a3 e8 fb 5b 8a bd d0 7d 43
a4 ee 80 f8 c8 8b 4e 50 5c dd 21 00 3b d0 bc cf
e2 d5 11 d4 7d 98 08 a7 16 7b 8c 56 44 ba 6d 53
ad 63 65 fd bf 3f 1f 4a a1 c5 d0 58 23 ae d1 47
80 74 f1 d0 b9 00 e5 1d 50 74 53 96 4b ce 59 50

sh$ hexdump -C ./api_iv_10
00000000  80 74 f1 d0 b9 00 e5 1d  50 74 53 96 4b ce 59 50  |.t......PtS.K.YP|
00000010

As a personal opinion, if you really have a lot of binary data processing, I would recommend to switch to some other language more data oriented (Python, ...)

I know there's the option -hex for openssl rand, but I need both versions: the binary and the hex string. In case you wonder, the hex string is used as the IV to encrypt some data, and the binary is used to prepend to that encrypted data. If I accept this answer, is there a nice way to convert the hex string to binary? — Luan Nguyen, Jul 16 '14 at 20:52
@LuanNguyen Are you certain that you can _store_ binary data containing `\x00` in a shell variable? As far as I can tell, the shell is confused by that end-of-string marker. In my bash: `V=$'\xFF\xAA\x00\xBB'; echo "${#V}"` display `2`. For the bash, this binary string is two bytes long. — Sylvain Leroux, Jul 16 '14 at 20:57
@LuanNguyen I add a reference to an other question showing how to convert from _hex to bytes_ using a perl one-liner ... and several words of warning about storing binary data in shell variables ;) Hope this helps. — Sylvain Leroux, Jul 16 '14 at 21:21
Very good explanation. How do I accept this one as an answer? — Luan Nguyen, Jul 17 '14 at 15:09

score 2 · Answer 2 · answered Jul 16 '14 at 16:34

2

Because the missing byte was an ASCII NUL '\0' '\x00'. The echo command stops printing its argument(s) when it comes across a null byte in each argument.

answered Jul 16 '14 at 16:34

Jonathan Leffler

730,956
141
904
1,278

Bash echo command with binary data?

2 Answers2

Linked