Divide an extremely long byte stream into smaller bytes

Question

So i need to unpack an extremely long byte stream (from USB) into 4 byte values.

Currently i got it working, but i feel there's a better way to do this.

Currently i got:

l=[]
for i in range(int(len(mybytes)/4)):
    l.append(struct.unpack_from('>i',mybytes,i*4))

So this feels like very resource expensive, and im doing this for 16k bytes A LOT.

I also feel like this has probably been asked before i just don't really know how to word it for searching

Answer 1

You could also try the array module which has the ability to load directly from binary data:

 import array
 arr = array.array("I",mybytes) # "I" stands for unsigned integer
 arr.byteswap() # only if you're reading endian coding different from your platform
 l = list(arr)

Answer 2

You can specify a size for the integers to unpack (Python 3.6+):

>>> import struct
>>> mybytes = bytes([1,2,3,4,5,6,7,8])
>>> struct.unpack(f'>2i',mybytes)
(16909060, 84281096)
>>> n = len(mybytes) // 4
>>> struct.unpack(f'>{n}i',mybytes) # Python 3.6+ f-strings
(16909060, 84281096)
>>> struct.unpack('>{}i'.format(n),mybytes) # Older Pythons
(16909060, 84281096)
>>> [hex(i) for i in _]
['0x1020304', '0x5060708']

Answer 3

Wrap it in a BytesIO object, then use iter to call its read method until it returns an empty bytes value.

>>> import io, struct
>>> bio = io.BytesIO(b'abcdefgh')
>>> int_fmt = struct.Struct(">i")
>>> list(map(int_fmt.unpack, iter(lambda: bio.read(4), b'')))
[(1633837924,), (1701209960,)]

You can tweak this to extract the single int value from each tuple, or switch to the from_bytes class method.

>>> bio = io.BytesIO(b'abcdefgh')
>>> list(map(lambda i: int.from_bytes(i, 'big'), iter(lambda: bio.read(4), b'')))
[1633837924, 1701209960]