171

I have a list of bytes as integers, which is something like

[120, 3, 255, 0, 100]

How can I write this list to a file as binary?

Would this work?

newFileBytes = [123, 3, 255, 0, 100]
# make file
newFile = open("filename.txt", "wb")
# write to file
newFile.write(newFileBytes)
Pavel
  • 5,374
  • 4
  • 30
  • 55
Aaron Hiniker
  • 3,923
  • 6
  • 21
  • 14

7 Answers7

170

This is exactly what bytearray is for:

newFileByteArray = bytearray(newFileBytes)
newFile.write(newFileByteArray)

If you're using Python 3.x, you can use bytes instead (and probably ought to, as it signals your intention better). But in Python 2.x, that won't work, because bytes is just an alias for str. As usual, showing with the interactive interpreter is easier than explaining with text, so let me just do that.

Python 3.x:

>>> bytearray(newFileBytes)
bytearray(b'{\x03\xff\x00d')
>>> bytes(newFileBytes)
b'{\x03\xff\x00d'

Python 2.x:

>>> bytearray(newFileBytes)
bytearray(b'{\x03\xff\x00d')
>>> bytes(newFileBytes)
'[123, 3, 255, 0, 100]'
abarnert
  • 354,177
  • 51
  • 601
  • 671
  • 1
    Nice use of builtin types. Just note that bytearray was added in 2.6, if you want to support legacy systems, it should be avoided. – Perkins Aug 21 '13 at 20:33
  • 7
    @Perkins: Sure, and you should avoid generator expressions if you need to work on 2.3, be careful with both `str.encode` and `struct.pack` if you need to work on 2.2. But 2.6 has been out for 5 years now; all three Ubuntu LTSs still in support, all three OS X versions in support, the previous major version of CentOS/RHEL, etc., all come with it built in. If you need to support 2.5 or 2.1 or 1.6 or whatever, you probably know… – abarnert Aug 21 '13 at 21:19
  • 4
    With Python 2 on Windows, I found that writing a `bytearray` still converts `\n` to `\r\n`, making it unsatisfactory for binary data, if the "b" flag is not passed when opening the file. – feersum Dec 11 '14 at 13:03
  • 7
    @feersum: Of course; that's what binary vs. text mode _means_ in 2.x. It doesn't matter what type your bytes come from. (In 3.x, of course, binary vs. text mode means that you write bytes vs. unicode, and the `\r\n` feature is part of the universal newlines options for text.) – abarnert Dec 13 '14 at 02:01
  • I'm not sure bytearray() is a good choice for file writing. You would need to limit the size to manageable chunks. Otherwise once your filesizes get too high you will run out of memory. – mckenzm May 02 '19 at 02:20
38

Use struct.pack to convert the integer values into binary bytes, then write the bytes. E.g.

newFile.write(struct.pack('5B', *newFileBytes))

However I would never give a binary file a .txt extension.

The benefit of this method is that it works for other types as well, for example if any of the values were greater than 255 you could use '5i' for the format instead to get full 32-bit integers.

Putnik
  • 5,925
  • 7
  • 38
  • 58
Mark Ransom
  • 299,747
  • 42
  • 398
  • 622
  • .txt is fine if you have some way to knowing that the data you are writing all falls inside the printable ascii range. However, you are correct I think in this case, since the example data includes non printable characters. – Perkins Aug 21 '13 at 20:30
  • 1
    @Perkins I didn't make the assumption that the values would even be less than 256 much less in the ASCII range. Even if they are, .txt files should be reserved for those that make sense to a human which never applies to binary data. – Mark Ransom Aug 21 '13 at 20:33
  • 1
    You're right, struct.pack is also the way to go if you are going to be writing data with values above 255, since neither bytearray nor chr can handle larger integer values. – Perkins Aug 21 '13 at 20:40
  • @Perkins: But `struct.pack` can't handle values above 255 either. Nothing can, because there is no meaningful thing for them to do. You will get a `struct.error` instead of a `ValueError`, but otherwise there's not much difference. (Well, in 3.x, I guess `chr` will actually give you a `UnicodeEncodeError` instead…) – abarnert Aug 21 '13 at 21:22
  • @abarnert, I didn't intend to pack them to bytes but to 4-byte integers. I've edited the answer to be clearer. – Mark Ransom Aug 21 '13 at 21:33
  • @MarkRansom: Given that he has "a list of bytes as integers", I don't think that's the output he wants. – abarnert Aug 21 '13 at 21:49
  • @abarnert how did I miss that little detail in the question? Thanks! – Mark Ransom Aug 21 '13 at 21:53
  • 2
    @MarkRansom: Well, this is still definitely a good solution to the more general problem of "I have a list of integers of some arbitrary but fixed size, how can I write them to a binary file?" and I can see people searching for that question and finding this one… – abarnert Aug 21 '13 at 22:22
  • struct.pack can support values larger than 256, just not with the "i" format, use 'l' instead, so struct.pack('l',257) works fine, because it handles the numbers as longs instead of ints. – Perkins Aug 21 '13 at 22:30
  • @Perkins: You're mixing up your `struct` formats. Or maybe you're mixing up your C types. `int` is at least 16 bits, so it can already handle values >256. `long` is at least as long as `int`; on many platforms (Win32, Win64, most 32-bit *nix) but not all (most 64-bit *nix) it's in fact exactly the same size. And at any rate, the `'i'` and `'l'` formats don't actually use `int` or `long` unless you specify native packing; they both explicitly use 32 bits. It's `B` that can't handle values larger than 255. (And it's 255, not 256, of course.) – abarnert Aug 21 '13 at 22:46
  • Right, that all makes sense, been a long time since I messed around with the `struct` module, and most of the C I use is on an embedded 8bit system with 16 bit longs, you are of course right when I actually take the time to think about it. – Perkins Aug 22 '13 at 07:24
  • 2
    struct.pack is the better answer; it is far more flexible than simply creating a bytearray. – Seth Jul 06 '14 at 13:53
16

To convert from integers < 256 to binary, use the chr function. So you're looking at doing the following.

newFileBytes=[123,3,255,0,100]
newfile=open(path,'wb')
newfile.write((''.join(chr(i) for i in newFileBytes)).encode('charmap'))
Perkins
  • 2,409
  • 25
  • 23
  • 1
    You must mean < 128. As python3 complains: UnicodeEncodeError: 'ascii' codec can't encode character '\x89' in position 0: ordinal not in range(128) – elig Oct 07 '18 at 13:18
  • 2
    No, I mean < 256, but the encoding should be `charmap` rather than `ascii`, and works in python2 as well as python3. The `ascii` encoding only works in python2. – Perkins Oct 07 '18 at 17:11
16

As of Python 3.2+, you can also accomplish this using the to_bytes native int method:

newFileBytes = [123, 3, 255, 0, 100]
# make file
newFile = open("filename.txt", "wb")
# write to file
for byte in newFileBytes:
    newFile.write(byte.to_bytes(1, byteorder='big'))

I.e., each single call to to_bytes in this case creates a string of length 1, with its characters arranged in big-endian order (which is trivial for length-1 strings), which represents the integer value byte. You can also shorten the last two lines into a single one:

newFile.write(''.join([byte.to_bytes(1, byteorder='big') for byte in newFileBytes]))
CrepeGoat
  • 2,315
  • 20
  • 24
9

You can use the following code example using Python 3 syntax:

from struct import pack
with open("foo.bin", "wb") as file:
  file.write(pack("<IIIII", *bytearray([120, 3, 255, 0, 100])))

Here is shell one-liner:

python -c $'from struct import pack\nwith open("foo.bin", "wb") as file: file.write(pack("<IIIII", *bytearray([120, 3, 255, 0, 100])))'
kenorb
  • 155,785
  • 88
  • 678
  • 743
1

Use pickle, like this: import pickle

Your code would look like this:

import pickle
mybytes = [120, 3, 255, 0, 100]
with open("bytesfile", "wb") as mypicklefile:
    pickle.dump(mybytes, mypicklefile)

To read the data back, use the pickle.load method

Raymond Mlambo
  • 73
  • 6
  • 8
    This does not produce a binary file of 5 bytes length, where the only content is 120, 3, 255, 0, 100. In a closed system, this may be acceptable though. – parvus Jun 14 '18 at 08:49
0

Convenient function to write array of int to a file,

def write_array(fname,ray):
    '''
    fname is a file pathname
    ray is an array of int
    '''
    print("write:",fname)
    EncodeInit()
    buffer = [ encode(z) for z in ray ]
    some = bytearray(buffer)
    immutable = bytes(some)
    with open(fname,"wb") as bfh:
        wc = bfh.write(immutable)
        print("wrote:",wrote)
    return wc

How to call the function,

write_array("data/filename",[1,2,3,4,5,6,7,8])

And wrap the following in a class for readable encode/decode:

Encode = {}
Decode = {}
def EncodeInit():
    '''
    Encode[] 0:62 as 0-9A-Za-z
    Decode[] 0-9A-Za-z as 0:62
    '''
    for ix in range( 0,10): Encode[ix] = ix+ord('0')
    for ix in range(10,36): Encode[ix] = (ix-10)+ord('A')
    for ix in range(36,62): Encode[ix] = (ix-36)+ord('a')
    for ix in range( 0,10): Decode[ix+ord('0')] = ix
    for ix in range(10,36): Decode[(ix-10)+ord('A')] = ix
    for ix in range(36,62): Decode[(ix-36)+ord('a')] = ix

def encode(x):
    '''
    Encode[] 0:62 as 0-9A-Za-z
    Otherwise '.'
    '''
    if x in Encode: return Encode[x]
    # else: error
    return ord('.')

def decode(x):
    '''
    Decode[] 0-9A-Za-z as 0:62
    Otherwise -1
    '''
    if x in Decode: return Decode[x]
    # else: error
    return -1
ChuckCottrill
  • 4,360
  • 2
  • 24
  • 42