Display progress bar for running python scripts

Question

I would like to display a progress bar when running a long python program. I researched online and found the function below.

import sys, time

def progress(count, total, status=''):
    bar_len = 60
    filled_len = int(round(bar_len * count / float(total)))

    percents = round(100.0 * count / float(total), 1)
    bar = '=' * filled_len + '-' * (bar_len - filled_len)

    sys.stdout.write('\r[%s] %s%s ...%s\r' % (bar, percents, '%', status))
    sys.stdout.flush()  

Then, I did the following:

total = 1000
i = 0
while i < total:
    progress(i, total, status='Doing very long job')
    # My long python program here   
    i += 1

When I try the above, it takes much longer time to run my python program. Is the above the right way to use the progress function and display the progress bar? Thanks for your inputs.

Answer 1

The extra time taken in the execution is due to frequent (1000x) console flushing. Please have a read on this question: Why is printing to stdout so slow? Can it be sped up?

to resolve this slowdown, I would recommend printing status and flushing every e.g. 10 or 20 iterations:

total = 1000
i = 0
while i < total:
    if i % 20 == 0:
        progress(i, total, status='Doing very long job')
    # My long python program here   
    i += 1

However, the best way forward would be to use existing libraries, e.g. tqdm (https://github.com/tqdm/tqdm). It is well optimized and offers cool features like nested progress bars.

Answer 2

If third party libraries are ok I would recommend using tqdm. Also when asking about performance it would be helpful if you reported timings.

One way to speed up performance would be to not call progress for every iteration of the loop. IO is expensive so constantly printing to the screen can really bog down a program, especially if you are bypassing buffering by using flush.

Answer 3

I wrote a simple progress bar code to show some thing like this:

[#############4------] 67.00%

And to be placed with a FOR or a WHILE, but you need to know the total size:

class Progresso:
def __init__(self, total, steps):

    assert steps < 101 and type(total) == int and type(steps) == int and total > steps

    self.count = 0
    self.total = total
    self.steps = steps
    self.passo = total // steps
    if total < steps *10:
        self.test = 1
    else:
        self.test = total // (steps * 10)
        print(self.prog(), end='\r')

def printa(self):
    self.count += 1
    if self.count % self.test == 0:
        print(self.prog(), end='\r')


def prog(self):
    passado = self.count // self.passo
    fut = self.steps - passado + 1
    resto = int((self.count % self.passo) / self.passo * 10)
    if resto == 0:
        centro = '-'*(fut -1)
    else:
        centro = str(resto) + '-'*(fut-2)
    return '['+'#'*passado + centro + '] ' +'{:.2f}%     '.format(self.count/self.total*100)


def fim(self):
    self.count = self.total
    print(self.prog())

You start before the FOR/WHILE like this:

p = Progresso(200000, 20)

Inside the FOR/WHILE you insert the p.printa() and after the FOR/WHILE you end it: p.fim().

Answer 4

A fairly simple change to your progress function will ensure it only prints and flushes when the progress bar to be printed has changed, by saving the previously printed bar in a function attribute (see https://www.python.org/dev/peps/pep-0232/):

import sys, time

def progress(count, total, status=''):
    # make sure the saved bar is initialized
    if not hasattr(progress,'lastbar'):
        progress.lastbar = ""
    bar_len = 60
    filled_len = int(round(bar_len * count / float(total)))

    percents = round(100.0 * count / float(total), 1)
    bar = '=' * filled_len + '-' * (bar_len - filled_len)
    # only print the bar if it is different from the previous bar
    if bar != progress.lastbar:
        sys.stdout.write('\r[%s] %s%s ...%s\r' % (bar, percents, '%', status))
        sys.stdout.flush()  
        progress.lastbar = bar

total = 1000
i = 0
while i < total:
    progress(i, total, status='Doing very long job')
    # My long python program here   
    i += 1

The above code only prints and flushes when the bar itself has changed - so 60 times rather than 1000 times - ignoring changes to the displayed percentage and status message - if these matter to you, you might want to store and compare the complete printed line, like this (but this will do more printing and more flushing):

import sys, time

def progress(count, total, status=''):
    if not hasattr(progress,'lastbar'):
        progress.lastbar = ""
    bar_len = 60
    filled_len = int(round(bar_len * count / float(total)))

    percents = round(100.0 * count / float(total), 1)
    bar = '=' * filled_len + '-' * (bar_len - filled_len)
    fullbar = '\r[%s] %s%s ...%s\r' % (bar, percents, '%', status)
    if fullbar != progress.lastbar:
        sys.stdout.write(fullbar)
        sys.stdout.flush()  
        progress.lastbar = fullbar

total = 1000
i = 0
while i < total:
    progress(i, total, status='Doing very long job')
    # My long python program here   
    i += 1

I'm sure more sophisticated methods could be used to build the 'only one in 20' logic into the progress function, too.

It's possible that building the bar is also an expensive part of the progress function - you could avoid this when the output isn't going to change by comparing the filled_len with the previous filled_lenand don't even build the string for the bar if the value hasn't changed, like this:

import sys, time

def progress(count, total, status=''):
    if not hasattr(progress,'lastlen'):
        progress.lastlen = 0
    bar_len = 60
    filled_len = int(round(bar_len * count / float(total)))
    if filled_len != progress.lastlen:
        percents = round(100.0 * count / float(total), 1)
        bar = '=' * filled_len + '-' * (bar_len - filled_len)
        fullbar = '\r[%s] %s%s ...%s\r' % (bar, percents, '%', status)
        sys.stdout.write(fullbar)
        sys.stdout.flush()  
        progress.lastlen = filled_len

total = 1000
i = 0
while i < total:
    progress(i, total, status='Doing very long job')
    # My long python program here   
    i += 1