Read numeric data from binary file using python

Question

I would like to read a binary data file that contains a header part (text) and then a numeric array. I can use f.read(block_size) to keep streaming in the header part, but what is the best way to read the numeric array?

In MatLab, I could do

fid = fopen(data_file_name, 'rb');
line = fread(fid, block_size, '*char'); 
data = fread(fid, 'long');

In Python, what I have done is

f = open(data_file_name, 'rb')
header = f.read(block_size)

and from here I do not know how to get to the numeric array.

Answer 1

You can use struct.unpack to unpack the numeric data.

e.g.

with open('file','rb') as fin:
    header = fin.read(header_size)
    data_str = fin.read(num_data_bytes)
    data_tuple = struct.unpack('100f',data_str)  #100 4-byte floats

Depending on the data, you can read it directly to a numpy array using numpy.fromfile. That function accepts an open file object, so you can read the header and then pass the open file object in so numpy can read the data. In this question, I asked about the details of reading binary data from a string into a numpy array. It's a slightly different problem, but much of the answer there applies to this as well (how to specify endianness, etc.)