In Python What is the common interface of readline, readlines, read, etc

Question

I'm trying to use the function I came up with so I can add a progress bar display for how much the file has been read but when I do this:

file_path = 'file'
f = open(file_path, 'rb')
file_size = os.path.getsize(file_path) 

def new_readline(self, size=-1):
    print(self.tell(), '/', file_size)

f.read = hook(new_readline, f.read)

If I try to read the lines of the the file with say f.readlines() it won't show anything, it only displays an output when I use f.readline().

even if I hook f.read I still get nothing.

So, what is a common method that all data reading method call (doesn't have to be a readline method, just something that they all call on self for) that I can hook onto?

There is no such common method. There is no reason to expect a common method, either. — user2357112, Sep 21 '21 at 12:01
(You're lucky you could even reassign `f.read` at all - most built-in types wouldn't let you do that.) — user2357112, Sep 21 '21 at 12:05
So I have to add a hook to all the read methods, oh well, fair enough, thank you for your response, I get that it's a strange question. I'm only having to resort to his because the python openstack SDK doesn't offer a way to read progress for image uploads (only downloads by streaming, chunk by chunk). — Thermatix, Sep 21 '21 at 12:11
Your replacement method doesn't actually read anything, and especially doesn't *return* the requested content. Why are you confused that using it leads to reading not providing the expected content? — MisterMiyagi, Sep 21 '21 at 12:16
@MisterMiyagi, It's not supposed to replace anything, it's meant to be called before the original method, but it still calls the original so it doesn't need to return anything, It's just a way to interact with the object each time `readline()` (in this instance) is called. The issue is I thought that `readlines()` used `readline()` internally, so If I add the hook there, then my hook would be called every time data is read, no matter how it happens, from the file the file-object represents. — Thermatix, Sep 21 '21 at 12:33
Piecing together a [mre] from this… it just works for both ``readlines`` and ``readline`` (though neither uses the hook). Can you provide a [mre]? Did you read from the same file object twice? — MisterMiyagi, Sep 21 '21 at 13:02
in the end, it doesn't really matter. I just wanted to be able to add the hook in once place and then have it activate no matter what because I don't know what method the open-stack SDK is using to open the file. So if I can just have it affect all file read operations from one place then I can guarantee my hook will be executed each time the file is read. — Thermatix, Sep 21 '21 at 13:06
There is no such "one place" in Python. File objects are built-ins – actual reads are done inside compiled code wrapping some system library. The methods available in the Python code are just a facade *around* that. — MisterMiyagi, Sep 21 '21 at 13:12

Thermatix · Accepted Answer · 2021-09-21T15:02:39.390

In the end I had to attach the hook to the read() method, but it works!

Any way, here's the full code for those that are interested:

from types import MethodType
import os

ESCAPE = {
    'c': { # colours
        'fg': { # foreground
            'b': '30', # black
            'r': '31', # red
            'g': '32', # green
            'y': '33', # yellow
            'u': '34', # blue
            'p': '35', # purple
            'c': '36', # cyan
            'w': '37', # white
        },
        'bg': { # background
            'b': '40', # black
            'r': '41', # red
            'g': '42', # green
            'y': '43', # yellow
            'u': '44', # blue
            'p': '45', # purple
            'c': '46', # cyan
            'w': '47', # white
        }
    },
    's': { # style
        'n': '0',  # none
        'b': '1',  # bold
        'u': '2',  # underline
        'n1': '3', # negative1
        'n2': '5', # negative2
    },
    't': '\033[{s};{fg};{bg}m', # template 
    'util': {
        'hide': '\033[?25l',
        'show': '\033[?25h',
        'go_up': '\033[{}A',
        'go_down': '\033[{}B',
        'erase_line': '\033[K',
    }
} 


# Open a file but attach an upload progress display if it was requested
def open_file(args, mode='r'):
    f = open(args.file_name, mode)
    if args.progress:
        f.size = os.path.getsize(args.file_name) 
        f.spinner = spinner(
            f"Uploading image({colour(s='b')}{args.file_name}{colour()}) \
as '{colour(s='b')}{args.image_name}{colour()}' -"
        )
        def read_progress(self, size):
            print(
        f"{next(self.spinner)}{next(progress(self.size, step=size, p=self.tell()))}", end='\r'
            )

        f.read = hook(f.read, read_progress)
    return f

# Attach a callback to be executed whenever this method is called
# Note, callback args must match method being hooked
def hook(oldfunc, hookfunk):
    def merged(self, *args, **kwargs):
        hookfunk(self, *args, **kwargs)
        return oldfunc(*args, **kwargs)
    return  MethodType(merged, oldfunc.__self__)

# A spinner with a message, use `next()` to get the next frame
def spinner(msg: str):
    template = f"{colour(fg='p', s='b')}{'{}'}{colour()} {msg}"
    while True:
        for spin in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏':
            yield template.format(spin)

# Generate a progress bar, use `next()` to get the next progress
def progress(end: int, max_col=80, step=1024, p=1):
    template = f"[{colour(fg='c', s='b')}{'{}'}{colour(fg='w')}{'{}'}{colour()}|{'{}'}]"
    while (p <= end):
        bar = '▇' * int(max_col * p/(end-1))
        togo = '▇' * int(max_col - len(bar))
        perc = "%6.2f %%" % (p/(end-1)*100)
        yield template.format(bar, togo, perc)
        p += step

# Set the colour of the next segment of text
def colour(fg='w', bg='b', s='n'):
    return ESCAPE['t'].format(
            s=ESCAPE['s'][s],
            fg=ESCAPE['c']['fg'][fg],
            bg=ESCAPE['c']['bg'][bg]
    )

P.s. I realise that it's not efficient to re-create the progress bar every time, but I don't know any other way to change the step value each time as the speed of upload is not constant and I still need it to work with next for use in other places.

In Python What is the common interface of readline, readlines, read, etc

1 Answers1