Writing a BytesIO object to a file, 'efficiently'

Question

So a quick way to write a BytesIO object to a file would be to just use:

with open('myfile.ext', 'wb') as f:
    f.write(myBytesIOObj.getvalue())
myBytesIOObj.close()

However, if I wanted to iterate over the myBytesIOObj as opposed to writing it in one chunk, how would I go about it? I'm on Python 2.7.1. Also, if the BytesIO is huge, would it be a more efficient way of writing by iteration?

Thanks

score 28 · Accepted Answer · edited Apr 15 '19 at 12:34

shutil has a utility that will write the file efficiently. It copies in chunks, defaulting to 16K. Any multiple of 4K chunks should be a good cross platform number. I chose 131072 rather arbitrarily because really the file is written to the OS cache in RAM before going to disk and the chunk size isn't that big of a deal.

import shutil

myBytesIOObj.seek(0)
with open('myfile.ext', 'wb') as f:
    shutil.copyfileobj(myBytesIOObj, f, length=131072)

BTW, there was no need to close the file object at the end. with defines a scope, and the file object is defined inside that scope. The file handle is therefore closed automatically on exit from the with block.

Awesome. Thanks so much. This works perfectly. And yes, I'll not bother closing my file pointer. However, my Bytes/String IO pointer, I want to close, in effect, discarding it and letting it go to the garbage collector. — Kalabaaz, Aug 20 '16 at 05:05

score 11 · Answer 2 · answered Oct 21 '20 at 07:33

Since Python 3.2 it's possible to use the BytesIO.getbuffer() method as follows:

from io import BytesIO
buf = BytesIO(b'test')
with open('path/to/file', 'wb') as f:
    f.write(buf.getbuffer())

This way it doesn't copy the buffer's content, streaming it straight to the open file.

Note: The StringIO buffer doesn't support the getbuffer() protocol (as of Python 3.9).

Before streaming the BytesIO buffer to file, you might want to set its position to the beginning:

buf.seek(0)

Writing a BytesIO object to a file, 'efficiently'

2 Answers2