What language is to binary, as Perl is to text?

Question

I am looking for a scripting (or higher level programming) language (or e.g. modules for Python or similar languages) for effortlessly analyzing and manipulating binary data in files (e.g. core dumps), much like Perl allows manipulating text files very smoothly.

Things I want to do include presenting arbitrary chunks of the data in various forms (binary, decimal, hex), convert data from one endianess to another, etc. That is, things you normally would use C or assembly for, but I'm looking for a language which allows for writing tiny pieces of code for highly specific, one-time purposes very quickly.

Any suggestions?

Answer 1

Things I want to do include presenting arbitrary chunks of the data in various forms (binary, decimal, hex), convert data from one endianess to another, etc. That is, things you normally would use C or assembly for, but I'm looking for a language which allows for writing tiny pieces of code for highly specific, one-time purposes very quickly.

Well, while it may seem counter-intuitive, I found erlang extremely well-suited for this, namely due to its powerful support for pattern matching, even for bytes and bits (called "Erlang Bit Syntax"). Which makes it very easy to create even very advanced programs that deal with inspecting and manipulating data on a byte- and even on a bit-level:

Since 2001, the functional language Erlang comes with a byte-oriented datatype (called binary) and with constructs to do pattern matching on a binary.

And to quote informIT.com:

(Erlang) Pattern matching really starts to get fun when combined with the binary type. Consider an application that receives packets from a network and then processes them. The four bytes in a packet might be a network byte-order packet type identifier. In Erlang, you would just need a single processPacket function that could convert this into a data structure for internal processing. It would look something like this:

processPacket(<<1:32/big,RestOfPacket>>) ->
    % Process type one packets
    ...
;
processPacket(<<2:32/big,RestOfPacket>>) ->
    % Process type two packets
    ...

So, erlang with its built-in support for pattern matching and it being a functional language is pretty expressive, see for example the implementation of ueencode in erlang:

uuencode(BitStr) ->
<< (X+32):8 || <<X:6>> <= BitStr >>.
uudecode(Text) ->
<< (X-32):6 || <<X:8>> <= Text >>.

For an introduction, see Bitlevel Binaries and Generalized Comprehensions in Erlang.You may also want to check out some of the following pointers:

• Parsing Binaries with erlang, lamers inside
• More File Processing with Erlang
• Learning Erlang and Adobe Flash format same time
• Large Binary Data is (not) a Weakness of Erlang
• Programming Efficiently with Binaries and Bit Strings
• Erlang bit syntax and network programming
• erlang, the language for network programming (1)
• Erlang, the language for network programming Issue 2: binary pattern matching
• An Erlang MIDI File Reader/Writer
• Erlang Bit Syntax
• Comprehending endianness
• Playing with Erlang
• Erlang: Pattern Matching Declarations vs Case Statements/Other
• A Stream Library using Erlang Binaries
• Bit-level Binaries and Generalized Comprehensions in Erlang
• Applications, Implementation and Performance Evaluation of Bit Stream Programming in Erlang

Answer 2

perl's pack and unpack ?

Answer 3

The Python bitstring module was written for this purpose. It lets you take arbitary slices of binary data and offers a number of different interpretations through Python properties. It also gives plenty of tools for constructing and modifying binary data.

For example:

>>> from bitstring import BitArray, ConstBitStream
>>> s = BitArray('0x00cf')                           # 16 bits long
>>> print(s.hex, s.bin, s.int)                       # Some different views
00cf 0000000011001111 207
>>> s[2:5] = '0b001100001'                           # slice assignment
>>> s.replace('0b110', '0x345')                      # find and replace
2                                                    # 2 replacements made
>>> s.prepend([1])                                   # Add 1 bit to the start
>>> s.byteswap()                                     # Byte reversal
>>> ordinary_string = s.bytes                        # Back to Python string

There are also functions for bit-wise reading and navigation in the bitstring, much like in files; in fact this can be done straight from a file without reading it into memory:

>>> s = ConstBitStream(filename='somefile.ext')
>>> hex_code, a, b = s.readlist('hex:32, uint:7, uint:13')
>>> s.find('0x0001')         # Seek to next occurence, if found
True
    

There are also views with different endiannesses as well as the ability to swap endianness and much more - take a look at the manual.

Answer 4

Take a look at python bitstring, it looks like exactly what you want :)

Answer 5

I'm using 010 Editor to view binary files all the time to view binary files. It's especially geared to work with binary files.

It has an easy to use c-like scripting language to parse binary files and present them in a very readable way (as a tree, fields coded by color, stuff like that).. There are some example scripts to parse zipfiles and bmpfiles.

Whenever I create a binary file format, I always make a little script for 010 editor to view the files. If you've got some header files with some structs, making a reader for binary files is a matter of minutes.

Answer 6

Any high-level programming language with pack/unpack functions will do. All 3 Perl, Python and Ruby can do it. It's matter of personal preference. I wrote a bit of binary parsing in each of these and felt that Ruby was easiest/most elegant for this task.

Answer 7

Why not use a C interpreter? I always used them to experiment with snippets, but you could use one to script something like you describe without too much trouble.

I have always liked EiC. It was dead, but the project has been resurrected lately. EiC is surprisingly capable and reasonably quick. There is also CINT. Both can be compiled for different platforms, though I think CINT needs Cygwin on windows.

Answer 8

Python's standard library has some of what you require -- the array module in particular lets you easily read parts of binary files, swap endianness, etc; the struct module allows for finer-grained interpretation of binary strings. However, neither is quite as rich as you require: for example, to present the same data as bytes or halfwords, you need to copy it between two arrays (the numpy third-party add-on is much more powerful for interpreting the same area of memory in several different ways), and, for example, to display some bytes in hex there's nothing much "bundled" beyond a simple loop or list comprehension such as [hex(b) for b in thebytes[start:stop]]. I suspect there are reusable third-party modules to facilitate such tasks yet further, but I can't point you to one...

Answer 9

Forth can also be pretty good at this, but it's a bit arcane.

Answer 10

Well, if speed is not a consideration, and you want perl, then translate each line of binary into a line of chars - 0's and 1's. Yes, I know there are no linefeeds in binary :) but presumably you have some fixed size -- e.g. by byte or some other unit, with which you can break up the binary blob.

Then just use the perl string processing on that data :)

Answer 11

If you're doing binary level processing, it is very low level and likely needs to be very efficient and have minimal dependencies/install requirements.

So I would go with C - handles bytes well - and you can probably google for some library packages that handle bytes.

Going with something like Erlang introduces inefficiencies, dependencies, and other baggage you probably don't want with a low-level library.