24

I want to use a buffered stream because I want to use a peek() method to peek ahead but use my stream with another method that expects a file-like object. (I'd use seek() but may have to handle piped-in I/O that doesn't support random access.)

But this test case fails:

AttributeError: 'file' object has no attribute '_checkReadable'

import sys
import io

srcfile = sys.argv[1]
with open(srcfile, 'rb') as f:
    fbuf = io.BufferedReader(f)
    print fbuf.read(20)

What's going on and how do I fix it? I thought BufferedReader was intended to buffer a stream. If so, why does the open() function not return something that's compatible with it?

codeforester
  • 39,467
  • 16
  • 112
  • 140
Jason S
  • 184,598
  • 164
  • 608
  • 970
  • It's interesting. Although we have same versions (2.7.x), we get slightly different error messages. – username Apr 17 '12 at 21:34
  • I found it out. There's a comment in io.py as """Method descriptions and default implementations are inherited from the C version however.""" So it's depending on c versions or OS. – username Apr 17 '12 at 21:42
  • @username: This is not OS-specific. My Python 2.6.7 complains about `_checkReadable` as well, while my 2.7.2 complains about `readable`. I can't find the commit right now, but this was probably changed this somewhere between 2.7.0 and 2.7.2. – Fred Foo Apr 17 '12 at 22:22
  • possible duplicate of [Making io.BufferedReader from sys.stdin in Python](http://stackoverflow.com/questions/6065173/making-io-bufferedreader-from-sys-stdin-in-python) – Jason S Apr 18 '12 at 01:32

3 Answers3

25

By the looks of your print statement, you're using Python 2. On that version, a file is not a valid argument to the BufferedReader constructor:

Under Python 2.x, this is proposed as an alternative to the built-in file object, but in Python 3.x it is the default interface to access files and streams. (1)

You should use io.open instead:

>>> f = io.open(".bashrc", "rb")

If you do this, there's no need to explicitly wrap it in a BufferedReader since that's exactly what io.open returns by default:

>>> type(f)
<type '_io.BufferedReader'>

See its docs for details; there's a buffering argument that controls the buffering.

In Python 3, open is io.open so the two I/O libraries have been merged back into one. It seems that io was added to Python 2.6 mostly for forward compatibility.

Blade
  • 984
  • 3
  • 12
  • 34
Fred Foo
  • 355,277
  • 75
  • 744
  • 836
  • 1
    there's no need to wrap for opening a file, but what if I use sys.stdin instead? – Jason S Apr 17 '12 at 23:11
  • @JasonS: then the [hack](http://stackoverflow.com/questions/6065173/making-io-bufferedreader-from-sys-stdin-in-python) that `username` pointed to is valid. Or `io.open("/dev/stdin")` if your platform has that file (but in either case stay clear of `sys.stdin`). – Fred Foo Apr 17 '12 at 23:26
6

You can set the amount of buffering in bytes by passing the buffering argument to open:

import sys

srcfile = sys.argv[1]
with open(srcfile, 'rb', buffering=30) as f:
    print(f.peek(30))
    print(f.read(20))

This is a BufferedReader:

>>> with open("test.txt", 'rb', buffering=30) as f:
...     type(f)
<class '_io.BufferedReader'>

Note that, by default, it's buffered to 1 - line buffered.

Gareth Latty
  • 86,389
  • 17
  • 178
  • 183
  • sure, that's fine, but eventually I'm going want to buffer around other sources of input that might not be buffered... or is everything just buffered by default? sys.stdin? network streams? – Jason S Apr 17 '12 at 21:23
  • Well, then you can do what you did and do ``BufferedReader(f)`` - as it's already a ``BufferedReader`` it will work. – Gareth Latty Apr 17 '12 at 21:28
  • 1
    This is the Python 3 solution; in Python 2 the OP should use `io.open`. – Fred Foo Apr 17 '12 at 21:46
  • @larsmans Ah, I didn't know this was a 3.x feature, you are correct. – Gareth Latty Apr 17 '12 at 21:49
  • Correction: this is actually a pretty valid solution also in Python 2, but `open` will return a `file`, not a `BufferedReader` in 2.x, so there's no `peek` method. – Fred Foo Apr 17 '12 at 21:52
1

In Python2, if you have to use file object as returned by open (or e.g. provided by some module routines which you cannot modify), you can use file descriptor obtained by fileno() for io.FileIO constructor, then pass io.FileIO object to io.BufferedReader constructor.

So, you sample code can be rewritten as follows:

import sys
import io

srcfile = sys.argv[1]
with open(srcfile, 'rb') as f:
    fio  = io.FileIO(f.fileno())
    fbuf = io.BufferedReader(fio)
    print fbuf.read(20)
majkelx
  • 151
  • 1
  • 4