How to create a hex dump of file containing only the hex characters without spaces in bash?

Question

How do I create an unmodified hex dump of a binary file in Linux using bash? The od and hexdump commands both insert spaces in the dump and this is not ideal.

Is there a way to simply write a long string with all the hex characters, minus spaces or newlines in the output?

See also: [How to print only the hex values from hexdump without line numbers or ASCII table?](https://stackoverflow.com/questions/15553493/how-to-print-only-the-hex-values-from-hexdump-without-line-numbers-or-ascii-tabl) — 0x90, Jul 26 '17 at 21:29

score 296 · Answer 1 · edited Sep 26 '20 at 05:16

296

xxd -p file

Or if you want it all on a single line:

xxd -p file | tr -d '\n'

edited Sep 26 '20 at 05:16

Giacomo1968

25,759
11
71
103

answered Apr 10 '10 at 20:27

mark4o

58,919
18
87
102

18

fyi To reverse the process: xxd -r -ps hexascii.txt file (it is ok with or without newlines) – Curtis Yallop May 27 '14 at 23:19

score 118 · Answer 2 · edited Jan 27 '17 at 13:52

118

Format strings can make hexdump behave exactly as you want it to (no whitespace at all, byte by byte):

hexdump -ve '1/1 "%.2x"'

1/1 means "each format is applied once and takes one byte", and "%.2x" is the actual format string, like in printf. In this case: 2-character hexadecimal number, leading zeros if shorter.

edited Jan 27 '17 at 13:52

John Gibb

10,603
2
37
48

answered Apr 10 '10 at 21:42

Michał Trybus

11,526
3
30
42

9

You need a `-v` or it will drop repeated bytes and replace them with an asterisk. – Dennis Williamson Apr 10 '10 at 22:35
2

I wonder if hexdump itself is able to append the newline (only) to the end of output.. (obvious appendage of `; echo` makes it impossible to use as bash alias) – mykhal Aug 22 '15 at 17:57
3

My alias: `alias to_hex="hexdump -ve '1/1 \"%.2x\"' && echo"` – devstuff Feb 16 '18 at 22:19
The iteration count and byte count default to one, so 1/1 may be omitted, leaving the `hexdump -ve '"%.2x"'` – Alex Che Sep 27 '18 at 10:16
@mykhal, it's possible, if you know the number of bytes in output. Say if you use hexdump to output only 13 first bytes: `hexdump -n 13 -e '13/1 "%.2x" "\n"'` – Alex Che Sep 27 '18 at 10:58
1

You can reverse this output back into a binary with `xdd -p -r < dump > file` – dagelf Nov 03 '21 at 09:04

score 28 · Answer 3 · answered Apr 10 '10 at 20:22

28

It seems to depend on the details of the version of od. On OSX, use this:

od -t x1 -An file |tr -d '\n '

(That's print as type hex bytes, with no address. And whitespace deleted afterwards, of course.)

answered Apr 10 '10 at 20:22

Donal Fellows

133,037
18
149
215

3

I would also add `-v` in this context, otherwise it skips repeats with `*`. – Ciro Santilli OurBigBook.com Mar 08 '19 at 11:24

score 12 · Answer 4 · answered Apr 10 '10 at 21:18

12

Perl one-liner:

perl -e 'local $/; print unpack "H*", <>' file

answered Apr 10 '10 at 21:18

Alan Haggai Alavi

72,802
19
102
127

1

Verified. Matches "xxd -p file | tr -d '\n'". – Curtis Yallop May 27 '14 at 22:56
1

fyi To reverse the process: perl -e 'local $/; print pack "H*", <>' file – Curtis Yallop May 27 '14 at 23:17
The "local $/" is unnecessary. – Curtis Yallop May 28 '14 at 15:39
Update to last comment: The "local $/" is unnecessary for "pack". For unpack, you need it but can alternatively put "undef $/". $/ is the line separator (default NL). undefined puts it into slurp-mode. So <> referenced in a string context pulls the whole binary file without parsing it into lines. – Curtis Yallop May 28 '14 at 15:54
Alternate form: perl -e 'print unpack "H*", join("", <>)' – Curtis Yallop May 28 '14 at 15:59
Alternate form 1 for reversal (hexascii to binary): perl -pe '$_=pack "H*", $_' file – Curtis Yallop May 28 '14 at 16:10
Alternate form 2 for reversal (hexascii to binary), removes any newlines (xxd -ps -r adds them): perl -pe 's/\n//g; $_=pack "H*", $_' file – Curtis Yallop May 28 '14 at 16:11

score 3 · Answer 5 · edited May 23 '17 at 12:34

3

The other answers are preferable, but for a pure Bash solution, I've modified the script in my answer here to be able to output a continuous stream of hex characters representing the contents of a file. (Its normal mode is to emulate hexdump -C.)

edited May 23 '17 at 12:34

Community

1
1

answered Apr 10 '10 at 23:54

Dennis Williamson

346,391
90
374
439

score 3 · Answer 6 · answered Jun 02 '21 at 20:24

You can use Python for this purpose:

python -c "print(open('file.bin','rb').read().hex())"

...where file.bin is your filename.

Explaination:

Open file.bin in rb (read binary) mode.
Read contents (returned as bytes object).
Use bytes method .hex(), which returns hex dump without spaces or new lines.
Print output.

score 2 · Answer 7 · answered Dec 17 '20 at 16:30

tldr;

$ od -t x1 -A n -v <empty.zip | tr -dc '[:xdigit:]' && echo 
504b0506000000000000000000000000000000000000
$

Explanation:

Use the od tool to print single hexadecimal bytes (-t x1) --- without address offsets (-A n) and without eliding repeated "groups" (-v) --- from empty.zip, which has been redirected to standard input. Pipe that to tr which deletes (-d) the complement (-c) of the hexadecimal character set ('[:xdigit:]'). You can optionally print a trailing newline (echo) as I've done here to separate the output from the next shell prompt.

References:

Mikko Rantalainen · Answer 8 · 2020-10-19T08:47:59.347

1

I think this is the most widely supported version (requiring only POSIX defined tr and od behavior):

cat "$file" | od -v -t x1 -A n | tr -d ' \n'

This uses od to print each byte as hex without address without skipping repeated bytes and tr to delete all spaces and linefeeds in the output. Note that not even the trailing linefeed is emitted here. (The cat is intentional to allow multicore processing where cat can wait for filesystem while od is still processing previously read part. Single core users may want replace that with < "$file" od ... to save starting one additional process.)

edited Oct 19 '20 at 08:47

answered Oct 19 '20 at 08:42

Mikko Rantalainen

14,132
10
74
112

No need to use `cat` here, just pass the filename to `od`. – Martin Jun 27 '22 at 18:21
I agree that historically connecting stdin is considered better and it's still the correct solution for single core CPUs. However, with modern systems with many idle cores in a single socket CPU, I think it's better to allow `cat` to read pipe buffer worth of input in parallel with the `od` at all times. This reduces the possibility of `od` stalling for reading the file. – Mikko Rantalainen Jun 28 '22 at 07:05
I did some microbenchmarks and it seems that using the `cat` at the start does allow using multiple CPU cores for the task in parallel but in practice, the intercommunication between the cores causes so much extra work that avoiding the use of `cat` at the start of pipeline is still faster – at least for locally accessible files. In case the file was on remote network drive with slow connection, using the `cat` here could improve performance. That said, the `od` can only handle about 4 MB/s so it will be the bottleneck in most cases. – Mikko Rantalainen Jun 29 '22 at 07:49

score 1 · Answer 9 · answered May 16 '21 at 09:41

This code produces a "pure" hex dump string and it runs faster than the all the other examples given. It has been tested on 1GB files filled with binary zeros, and all linefeeds. It is not data content dependent and reads 1MB records instead of lines.

perl -pe 'BEGIN{$/=\1e6} $_=unpack "H*"'

Dozens of timing tests show that for 1GB files, these other methods below are slower. All tests were run writing output to a file which was then verified by checksum. Three 1GB input files were tested: all bytes, all binary zeros, and all LFs.

hexdump -ve '1/1 "%.2x"'                    #  ~10x  slower
od -v -t x1 -An | tr -d "\n "               #  ~15x  slower
xxd -p | tr -d \\n                          #   ~3x  slower
perl -e 'local \$/; print unpack "H*", <>'  # ~1.5x  slower
- this also slurps the whole file into memory

To reverse the process:

perl -pe 'BEGIN{$/=\1e6} $_=pack "H*",$_'

How to create a hex dump of file containing only the hex characters without spaces in bash?

9 Answers9

tldr;

Explanation:

References:

Linked

Related