Python writing binary files, bytes

Question

Python 3. I'm using QT's file dialog widget to save PDFs downloaded from the internet. I've been reading the file using 'open', and attempting to write it using the file dialog widget. However, I've been running into a"TypeError: '_io.BufferedReader' does not support the buffer interface" error.

Example code:

with open('file_to_read.pdf', 'rb') as f1: 
    with open('file_to_save.pdf', 'wb') as f2:
        f2.write(f1)

This logic works properly with text files when not using the 'b' designator, or when reading a file from the web, like with urllib or requests. These are of the 'bytes' type, which I think I need to be opening the file as. Instead, it's opening as a Buffered Reader. I tried bytes(f1), but get "TypeError: 'bytes' object cannot be interpreted as an integer." Any ideaas?

'list' evidentally does not support the buffer interface either. — Turtles Are Cute, May 19 '13 at 02:31

dawg · Accepted Answer · 2016-04-22T15:06:41.300

19

If your intent is to simply make a copy of the file, you could use shutil

>>> import shutil
>>> shutil.copyfile('file_to_read.pdf','file_to_save.pdf')

Or if you need to access byte by byte, similar to your structure, this works:

>>> with open('/tmp/fin.pdf','rb') as f1:
...    with open('/tmp/test.pdf','wb') as f2:
...       while True:
...          b=f1.read(1)
...          if b: 
...             # process b if this is your intent   
...             n=f2.write(b)
...          else: break

But byte by byte is potentially really slow.

Or, if you want a buffer that will speed this up (without taking the risk of reading an unknown file size completely into memory):

>>> with open('/tmp/fin.pdf','rb') as f1:
...    with open('/tmp/test.pdf','wb') as f2:
...       while True:
...          buf=f1.read(1024)
...          if buf: 
...              for byte in buf:
...                 pass    # process the bytes if this is what you want
...                         # make sure your changes are in buf
...              n=f2.write(buf)
...          else:
...              break

With Python 2.7+ or 3.1+ you can also use this shortcut (rather than using two with blocks):

with open('/tmp/fin.pdf','rb') as f1,open('/tmp/test.pdf','wb') as f2:
    ...

edited Apr 22 '16 at 15:06

answered May 19 '13 at 03:18

dawg

98,345
23
131
206

Thank you - your 2nd and 3rd solutions both worked. (Can't use copyfile due to the way QT's save dialog works) Simply using .read() on the bufferedReader to convert to bytes seemed work as well - thought I'd tried that. Learned some new stuff from your examples. – Turtles Are Cute May 19 '13 at 14:13
1

Be careful, the variable name `bytes` may collide with the Python built-in type `bytes` which represent binary data in Python 3.x! (In Python 2.7 it is just an alias of `str`) – minmaxavg Apr 18 '16 at 04:41
@minmaxavg: you are right, and I will change when I get a chance. Thanks! – dawg Apr 20 '16 at 20:55
1

[`shutil.copyfileobj()` wraps the `while`-loop](https://github.com/python/cpython/blob/d60b71192179fa6b241d93789f7fb75b329a4cd1/Lib/shutil.py#L70-L76). See how it is used in [`shutil.copyfile()`](https://github.com/python/cpython/blob/d60b71192179fa6b241d93789f7fb75b329a4cd1/Lib/shutil.py#L114-L116). To copy the file metadata too, you could use `shutil.copy2()`. If you want [to access a file as a buffer; you could use `mmap` (it works even if the whole file doesn't fit in memory)](http://stackoverflow.com/a/20014805/4279) – jfs Apr 22 '16 at 18:54
@J.F.Sebastian ..but the virtual address space is limited (especially for 32-bit systems) and `mmap` probably isn't portable (what is `mmap` for Windows?). – minmaxavg May 18 '16 at 23:39
1

@minmaxavg: 1- even my phone is 64-bit. 2- Yes, there is `mmap` on Windows. Read the official Python docs for `mmap` module. – jfs May 19 '16 at 00:06

score 5 · Answer 2 · answered May 19 '13 at 12:53

5

It really doesn't make sense to write a file in another file. What you want is to write the contents of f1 in f2. You get the contents with f1.read(). So you have to do this:

with open('file_to_read.pdf', 'rb') as f1: 
    with open('file_to_save.pdf', 'wb') as f2:
        f2.write(f1.read())

answered May 19 '13 at 12:53

Kritzefitz

2,644
1
20
35

This is what the current solution looks like. – Turtles Are Cute May 19 '13 at 14:56
5

While this works, the entire file is read into memory before it is written -- not very memory friendly. As stated in the Python [docs](http://docs.python.org/2/tutorial/inputoutput.html#methods-of-file-objects) `it’s your problem if the file is twice as large as your machine’s memory` – May 19 '13 at 20:10

score 5 · Answer 3 · answered Dec 17 '16 at 14:46

5

learned from python cookbook

from functools import partial

with open(fpath, 'rb') as f, open(target_fpath, 'wb') as target_f: 
    for _bytes in iter(partial(f.read, 1024), ''):
        target_f.write(_bytes)

partial(f.read, 1024) returns a function, read the binary file 1024 bytes at every turn. iter will end when meet a blank string ''.

answered Dec 17 '16 at 14:46

WeizhongTu

6,124
4
37
51

I got an infinite loop with this code, probably the EOF is not recognized correctly. Answer from @dawg works fine as expected. Learned just further reading, use ` b'' ` as EOF ... – Erich Kuester Feb 17 '21 at 14:52

Python writing binary files, bytes

3 Answers3

Linked