How do I use tqdm to show progress bars when using read_csv in a Jupiter notebook using jupyterlab

Question

I want to read a huge csv using read_csv and pandas and I want to show a progress bar since it is taking too long. Is there a way to do this? I have just seen examples with loops.

Answer 1

Yes. You could abuse any of the number of arguments that accept a callable and call it at each row:

from tqdm.auto import tqdm

with tqdm() as bar:
    # do not skip any of the rows, but update the progress bar instead
    pd.read_csv('data.csv', skiprows=lambda x: bar.update(1) and False)

If you use Linux, you can get the total number of lines to get a more meaningful progress bar:

from tqdm.auto import tqdm

lines_number = !cat 'data.csv' | wc -l

with tqdm(total=int(lines_number[0])) as bar:
    pd.read_csv('data.csv', skiprows=lambda x: bar.update(1) and False)

But if you do not like for-loops, you may also dislike context managers. You could get away with:

def none_but_please_show_progress_bar(*args, **kwargs):
    bar = tqdm(*args, **kwargs)

    def checker(x):
        bar.update(1)
        return False

    return checker

pd.read_csv('data.csv', skiprows=none_but_please_show_progress_bar())

But I find it less stable - I do recommend to use the context manager based approach.