Read/Write Python List from/to Binary file

Question

According to Python Cookbook, below is how to write a list of tuple into binary file:

from struct import Struct
def write_records(records, format, f):
    '''
    Write a sequence of tuples to a binary file of structures.
    '''
    record_struct = Struct(format)
    for r in records:
        f.write(record_struct.pack(*r))

# Example
if __name__ == '__main__':
    records = [ (1, 2.3, 4.5),
                (6, 7.8, 9.0),
                (12, 13.4, 56.7) ]
    with open('data.b', 'wb') as f:
        write_records(records, '<idd', f)

And it works well. For reading (large amount of binary data), the author recommended the following:

>>> import numpy as np
>>> f = open('data.b', 'rb')
>>> records = np.fromfile(f, dtype='<i,<d,<d')
>>> records
array([(1, 2.3, 4.5), (6, 7.8, 9.0), (12, 13.4, 56.7)],
dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '<f8')])
>>> records[0]
(1, 2.3, 4.5)
>>> records[1]
(6, 7.8, 9.0)
>>>

It is also good, but this record is not a normal numpy array. For instance, type(record[0]) will return <type 'numpy.void'>. Even worse, I cannot extract the first column using X = record[:, 0].

Is there a way to efficiently load list(or any other types) from binary file into a normal numpy array? Thx in advance.

Answer 1

In [196]: rec = np.fromfile('data.b', dtype='<i,<d,<d')
In [198]: rec
Out[198]: 
array([( 1,   2.3,   4.5), ( 6,   7.8,   9. ), (12,  13.4,  56.7)], 
      dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '<f8')])

This is a 1d structured array

In [199]: rec['f0']
Out[199]: array([ 1,  6, 12], dtype=int32)
In [200]: rec.shape
Out[200]: (3,)
In [201]: rec.dtype
Out[201]: dtype([('f0', '<i4'), ('f1', '<f8'), ('f2', '<f8')])

Note that its tolist looks identical to your original records:

In [202]: rec.tolist()
Out[202]: [(1, 2.3, 4.5), (6, 7.8, 9.0), (12, 13.4, 56.7)]
In [203]: records
Out[203]: [(1, 2.3, 4.5), (6, 7.8, 9.0), (12, 13.4, 56.7)]

You could create a 2d array from either list with:

In [204]: arr2 = np.array(rec.tolist())
In [205]: arr2
Out[205]: 
array([[  1. ,   2.3,   4.5],
       [  6. ,   7.8,   9. ],
       [ 12. ,  13.4,  56.7]])
In [206]: arr2.shape
Out[206]: (3, 3)

There are other ways of converting a structured array to 'regular' array, but this is simplest and most consistent.

The tolist of a regular array uses nested lists. The tuples in the structured version are intended to convey a difference:

In [207]: arr2.tolist()
Out[207]: [[1.0, 2.3, 4.5], [6.0, 7.8, 9.0], [12.0, 13.4, 56.7]]

In the structured array the first field is integer. In the regular array the first column is same as the others, float.

If the binary file contained all floats, you could load it as a 1d of floats and reshape

In [208]: with open('data.f', 'wb') as f:
     ...:         write_records(records, 'ddd', f)
In [210]: rec2 = np.fromfile('data.f', dtype='<d')
In [211]: rec2
Out[211]: array([  1. ,   2.3,   4.5,   6. ,   7.8,   9. ,  12. ,  13.4,  56.7])

But to take advantage of any record structure in the binary file, you have load by records as well, which means structured array:

In [213]: rec3 = np.fromfile('data.f', dtype='d,d,d')
In [214]: rec3
Out[214]: 
array([(  1.,   2.3,   4.5), (  6.,   7.8,   9. ), ( 12.,  13.4,  56.7)], 
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])