Parsing 6 byte block of hex data and storing decimal values in data frame

Question

I am trying to use python to parse data from a .bin file that is stored in hexadecimal format. Unfortunately it's not as simple as converting the hex to decimal to get the data out.

The file contains 'pages' of 300 measures where a 'measure' is a 6 byte block of hex data containing the values of 6 variables: X axis, Y axis, Z axis, light meter, button status and a reserved space (i.e. empty).

Each axis covers 12 bits (i.e. 1.5 bytes), the light covers 10 bits, and the button status/spare are 1 bit each.

For example:

Raw data (6 bytes): F4 1F 9E 08 20 00 (repeated 300x per 'page')

X axis (12bits) = F4 1

Y axis (12bits) = F 9E

Z axis (12bits) = 08 2

light (10bits) = 0 0 (and first two bits of final 0)

button (1bit) = 2nd to last bit of final 0

reserved (1bit) = last bit of final 0

How would I go about parsing the file, splitting up the blocks into their variables and converting the hex to decimal? Aiming to have a data frame of the decimal values, with the variables as columns and each measure (i.e. 6 byte blocks) as a row.

Past the usual open/read code below I am not sure how best to approach this.

with open(<PATH>, 'r') as f:
    data = f.read()

Answer 1

you can try

with open(...) as f:
    for line in f:
        # remove all white spaces
        hex = line.replace(" ", "")

        # convert fist 12 bits into int
        x = int(hex[0:3], 16)

        # second
        y = int(hex[3:6], 16)

        # third
        z = int(hex[6:9], 16)

        # convert last 12 bits from HEX to BIN (add padding to make sure you always have 12 figures)
        # get first 10 bits and convert to int
        light = int('{:012b}'.format(int(hex[9:12],16))[0:10],2)
        # get bit 11
        button = int('{:012b}'.format(int(hex[9:12],16))[10:11],2)
        # get last bit
        reserved = int('{:012b}'.format(int(hex[9:12],16))[11:12],2)

Answer 2

The best method to read the binary data into bits would probably be numpy.fromfile, and then followed by np.unpackbits:

uint_arr = np.fromfile(<PATH>, dtype = "uint8")
bit_arr = np.unpackbits(uint_arr)

bit_arr will be an array of 1s and 0s. From there, we want to divide the bits into their 6 byte counterparts with reshape:

bit_arr = bit_arr.reshape(-1, 48)

Answer 3

Firstly I'm assuming your input file contains binary data and not character data (i.e. hex characters).

What makes this non-trivial is that the file format is not aligned on byte boundaries (3x 12 bits then 1x 10 bits then 2x 1 bit) and so some bit twiddling is needed.

If we examine the format for a single measure we have this:

#   0xF4      0x1F      0x9E      0X08      0x20      0x00
# --------- --------- --------- --------- --------- ------------
# 0xF  0x4  0x1  0xF  0x9  0xE  0x0  0x8  0x2  0x0  0x0  0x0
# 1111 0100 0001 1111 1001 1110 0000 1000 0010 0000 0000 00 0 0

The alignment we want is:

#     x-axis         y-axis        z-axis         light     b r
# -------------- -------------- -------------- ------------ - -
# 1111 0100 0001 1111 1001 1110 0000 1000 0010 0000 0000 00 0 0

Therefore to get the interpretation you want we need to do some bit manipulation. The below shows how to do this long-hand (there are likely APIs to help made this simpler):

MEASURE_SIZE_BYTES = 6
MEASURES_PER_PAGE = 300
PAGE_SIZE_BYTES = MEASURE_SIZE_BYTES * MEASURES_PER_PAGE

def readPage(data):
    measures = []
    for measure in xrange(0, MEASURES_PER_PAGE):
        i = measure * MEASURE_SIZE_BYTES
        xAxis = (data[i] << 4) + (data[i + 1] >> 4)
        yAxis = ((data[i + 1] & 0xF) << 8) + (data[i + 2])
        zAxis = (data[i + 3] << 4) + (data[i + 4] >> 4)
        light = ((data[i + 4] & 0xF) << 8) + (data[i + 5] & 0xFC)
        button = (data[i + 5] & 0x2) >> 1
        reserved = (data[i + 5] & 0x1) >> 0
        measures.append({'xAxis': hex(xAxis), 'yAxis': hex(yAxis), 
                         'zAxis': hex(zAxis), 'light': hex(light), 
                         'button': hex(button), 'reserved': hex(reserved)})
    return measures

with open("data.bin", 'rb') as f:
    pages = []
    pageBytes = bytearray(f.read(PAGE_SIZE_BYTES))
    while len(pageBytes) == PAGE_SIZE_BYTES:
        pages.append(readPage(pageBytes))
        pageBytes = bytearray(f.read(PAGE_SIZE_BYTES))
    print(pages)

Produces (for a single page/measure):

[[{'reserved': '0x0', 'yAxis': '0xf9e', 'light': '0x0', 'button': '0x0', 'zAxis': '0x82', 'xAxis': '0xf41'}]]

To show the results in base10:

[[{'button': 0, 'light': 0, 'reserved': 0, 'xAxis': 3905, 'yAxis': 3998, 'zAxis': 130}]]