Reading and interpreting data from a binary file in Python

Question

I want to read a file byte by byte and check if the last bit of each byte is set:

#!/usr/bin/python

def main():
    fh = open('/tmp/test.txt', 'rb')
    try:
        byte = fh.read(1)
        while byte != "":
            if (int(byte,16) & 0x01) is 0x01:
                print 1
            else:
                print 0
            byte = fh.read(1)
    finally:
        fh.close

    fh.close()

if __name__ == "__main__":
        main()

The error I get is:

Traceback (most recent call last):
  File "./mini_01.py", line 21, in <module>
    main()
  File "./mini_01.py", line 10, in main
    if (int(byte,16) & 0x01) is 0x01:
ValueError: invalid literal for int() with base 16: '\xaf'

Anyone an idea? I didn't succeed using the struct and the binascii modules.

Read http://stackoverflow.com/questions/306313/python-is-operator-behaves-unexpectedly-with-integers before you get *this* working and open a dupe of *that* question ;) — , Oct 15 '10 at 14:31

Scott Griffiths · Answer 1 · 2010-10-15T18:51:26.017

Try using the bytearray type (Python 2.6 and later), it's much better suited to dealing with byte data. Your try block would be just:

ba = bytearray(fh.read())
for byte in ba:
    print byte & 1

or to create a list of results:

low_bit_list = [byte & 1 for byte in bytearray(fh.read())]

This works because when you index a bytearray you just get back an integer (0-255), whereas if you just read a byte from the file you get back a single character string and so need to use ord to convert it to an integer.

If your file is too big to comfortably hold in memory (though I'm guessing it isn't) then an mmap could be used to create the bytearray from a buffer:

import mmap
m = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
ba = bytearray(m)

+1 for the large file solution, I was saved from much hair-tearing. You sir are a gentleman and a scholar. — brichins, Sep 11 '13 at 15:16

nmichaels · Accepted Answer · 2010-10-15T15:15:23.877

9

You want to use ord instead of int:

if (ord(byte) & 0x01) == 0x01:

edited Oct 15 '10 at 15:15

answered Oct 15 '10 at 14:27

nmichaels

49,466
12
107
135

6

and you really should compare integers with `==`, not by their identities! – Scott Griffiths Oct 15 '10 at 14:45
Good point. I thought about it, but didn't change it when I pasted that...fixed. – nmichaels Oct 15 '10 at 15:15

score 5 · Answer 3 · answered Oct 15 '10 at 19:22

One way:

import array

filebytes= array.array('B')
filebytes.fromfile(open("/tmp/test.txt", "rb"))
if all(i & 1 for i in filebytes):
    # all file bytes are odd

Another way:

fobj= open("/tmp/test.txt", "rb")

try:
    import functools
except ImportError:
    bytereader= lambda: fobj.read(1)
else:
    bytereader= functools.partial(fobj.read, 1)

if all(ord(byte) & 1 for byte in iter(bytereader, '')):
    # all bytes are odd
fobj.close()

Reading and interpreting data from a binary file in Python

3 Answers3

Linked