0

If passing a buffer to TextIOWrapper (or maybe for completeness, other cases as well?), what are the possible consequences (in terms of memory usage, exceptions, other?) of the buffer not respecting the n parameter passed to its read method?

Say in the below example, it all appears to work fine:

from io import IOBase, TextIOWrapper

bytes_iter = (
    b'some',
    b'bytes',
    b'maybe a lot' + b'-' * 10000
)
it = iter(bytes_iter)
class FileLikeObj(IOBase):
    def readable(self):
        return True
    def read(self, size=-1):
        print('Asked for:', size)
        return next(it, b'')

text_io = TextIOWrapper(FileLikeObj(), encoding='utf-8', newline='')
for line in text_io:
   print('Found a line')

even though it returns more than the 8192 bytes that (for me) are always asked for, but is this fine in general?

This is inspired by/related to the answer at https://stackoverflow.com/a/70639580/1319998, and I'm trying to determine if the buffer should respect n or not

Michal Charemza
  • 25,940
  • 14
  • 98
  • 165
  • 1
    The consequences of returning *fewer* than n bytes might be even more interesting. In particular, if you include two `b''` in a row, then you return zero bytes twice in a row, and that seems to make it believe that it has reached the end and it stops. – Kelly Bundy Jan 09 '22 at 17:51
  • @KellyBundy For some reason I thought it was `b''` just the _once_ to indicate the end. Good to know! – Michal Charemza Jan 09 '22 at 20:54
  • A single one seems to have another effect, namely to signify the end of a "line". Try that along with `print('Found a line of length', len(line))`. – Kelly Bundy Jan 09 '22 at 21:10
  • 1
    [Demo](https://tio.run/##XZBBa8MwDIXv/hW62d4SWNmlFHLoDoXCIDAKOxanUVqvjh1sha3785mdhGydTtHzy9MndTe6OPu87vwwNN61oB3otnOeYF@@qIAZHPCL9uW7V12HnrHqRhiOmtBDAYJBrIoH1yLP5mZ0LN3y0apbhaDAOOLwGIWcwwOsnmIxyTTFuJQqfgdIdjIqBNhpg6/6imX1ISYquRlDa2zAo6pVZVAENM2sp/JIvbdw8D3eeUdfBkF/Y5Gv7n4IvUkUNi4sNGUJXi7PndeWBN@GK9bQOL/hU0gG/G0chXWSDFoxJUn5n2XSGaM44BgPXdzfVvzdU2aA9uRqbc8F76nJ1zHc4qfRFovEFREgNaAtzIHjLjPnzvW2TtdOFtckrjNdZsAkSjkMPw) with some more printing. – Kelly Bundy Jan 09 '22 at 21:11

0 Answers0