How to write a download progress indicator in Python?

Question

I am writing a little application to download files over http (as, for example, described here).

I also want to include a little download progress indicator showing the percentage of the download progress.

Here is what I came up with:

    sys.stdout.write(rem_file + "...")    
    urllib.urlretrieve(rem_file, loc_file, reporthook=dlProgress)

    def dlProgress(count, blockSize, totalSize):
      percent = int(count*blockSize*100/totalSize)
      sys.stdout.write("%2d%%" % percent)
      sys.stdout.write("\b\b\b")
      sys.stdout.flush()

Output: MyFileName... 9%

Any other ideas or recommendations to do this?

One thing that's somewhat annoying is the blinking cursor in the terminal on the first digit of the percentage. Is there a way to prevent this? Is there a way to hide the cursor?

EDIT:

Here a better alternative using a global variable for the filename in dlProgress and the '\r' code:

    global rem_file # global variable to be used in dlProgress

    urllib.urlretrieve(rem_file, loc_file, reporthook=dlProgress)

    def dlProgress(count, blockSize, totalSize):
      percent = int(count*blockSize*100/totalSize)
      sys.stdout.write("\r" + rem_file + "...%d%%" % percent)
      sys.stdout.flush()

Output: MyFileName...9%

And the cursor shows up at the END of the line. Much better.

`global rem_file` makes sense only inside a function where you bind it to a new object `rem_file = ...` otherwise (if you only read its value) `global rem_file` is not necessary. — jfs, Dec 14 '08 at 13:04
You can also do a /r + flush() on standard out. I am just guessing this is running in windows based on the terminal behavior you mention. — meawoppl, May 02 '14 at 20:53
related http://stackoverflow.com/questions/15644964/python-progress-bar-and-downloads — matt wilkie, Mar 22 '15 at 06:35

score 18 · Accepted Answer · edited Jun 20 '20 at 09:12

There's a text progress bar library for python at http://pypi.python.org/pypi/progressbar/2.2 that you might find useful:

This library provides a text mode progressbar. This is tipically used to display the progress of a long running operation, providing a visual clue that processing is underway.

The ProgressBar class manages the progress, and the format of the line is given by a number of widgets. A widget is an object that may display diferently depending on the state of the progress. There are three types of widget: - a string, which always shows itself; - a ProgressBarWidget, which may return a diferent value every time it's update method is called; and - a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it expands to fill the remaining width of the line.

The progressbar module is very easy to use, yet very powerful. And automatically supports features like auto-resizing when available.

how to have multiple progress bar in terminal if multiple downloads are intimated in separate threads ? — Ciasto piekarz, Jul 05 '14 at 06:34

score 15 · Answer 2 · answered Sep 09 '08 at 04:21

You might also try:

sys.stdout.write("\r%2d%%" % percent)
sys.stdout.flush()

Using a single carriage return at the beginning of your string rather than several backspaces. Your cursor will still blink, but it'll blink after the percent sign rather than under the first digit, and with one control character instead of three you may get less flicker.

tstone2077 · Answer 3 · 2016-10-17T13:34:20.383

7

For what it's worth, here's the code I used to get it working:

from urllib import urlretrieve
from progressbar import ProgressBar, Percentage, Bar

url = "http://......."
fileName = "file"
pbar = ProgressBar(widgets=[Percentage(), Bar()])
urlretrieve(url, fileName, reporthook=dlProgress)

def dlProgress(count, blockSize, totalSize):
    pbar.update( int(count * blockSize * 100 / totalSize) )

edited Oct 17 '16 at 13:34

answered Mar 15 '16 at 21:52

tstone2077

514
4
13

1

This only gives me: pbar.update( int(count * blocksize * 100 / totalSize) ) NameError: global name 'blocksize' is not defined – rm- Oct 14 '16 at 08:19
it was a typo. It should have been the input parameter. I've updated the answer to reflect that. Thanks! – tstone2077 Oct 17 '16 at 13:35

score 4 · Answer 4 · answered Sep 09 '08 at 04:14

If you use the curses package, you have much greater control of the console. It also comes at a higher cost in code complexity and is probably unnecessary unless you are developing a large console-based app.

For a simple solution, you can always put the spinning wheel at the end of the status messge (the sequence of characters |, \, -, / which actually looks nice under blinking cursor.

score 1 · Answer 5 · answered Jan 02 '13 at 14:01

I used this code:

url = (<file location>)
file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
meta = u.info()
file_size = int(meta.getheaders("Content-Length")[0])
print "Downloading: %s Bytes: %s" % (file_name, file_size)

file_size_dl = 0
block_sz = 8192
while True:
    buffer = u.read(block_sz)
    if not buffer:
        break

    file_size_dl += len(buffer)
    f.write(buffer)
    status = r"%10d [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
    status = status + chr(8)*(len(status)+1)
    print status,

f.close()

score 1 · Answer 6 · answered Apr 18 '17 at 14:43

def download_progress_hook(count, blockSize, totalSize):
  """A hook to report the progress of a download. This is mostly intended for users with slow internet connections. Reports every 5% change in download progress.
  """
  global last_percent_reported
  percent = int(count * blockSize * 100 / totalSize)

  if last_percent_reported != percent:
    if percent % 5 == 0:
      sys.stdout.write("%s%%" % percent)
      sys.stdout.flush()
    else:
      sys.stdout.write(".")
      sys.stdout.flush()

    last_percent_reported = percent

urlretrieve(url, filename, reporthook=download_progress_hook)

score 0 · Answer 7 · answered Oct 14 '12 at 08:31

0

Thats how I did this could help you: https://github.com/mouuff/MouDownloader/blob/master/api/download.py

answered Oct 14 '12 at 08:31

Arnaud Aliès

1,079
13
26

score 0 · Answer 8 · answered Feb 10 '15 at 04:38

Late to the party, as usual. Here's an implementation that supports reporting progress, like the core urlretrieve:

import urllib2

def urlretrieve(urllib2_request, filepath, reporthook=None, chunk_size=4096):
    req = urllib2.urlopen(urllib2_request)

    if reporthook:
        # ensure progress method is callable
        if hasattr(reporthook, '__call__'):
            reporthook = None

        try:
            # get response length
            total_size = req.info().getheaders('Content-Length')[0]
        except KeyError:
            reporthook = None

    data = ''
    num_blocks = 0

    with open(filepath, 'w') as f:
        while True:
            data = req.read(chunk_size)
            num_blocks += 1
            if reporthook:
                # report progress
                reporthook(num_blocks, chunk_size, total_size)
            if not data:
                break
            f.write(data)

    # return downloaded length
    return len(data)

score 0 · Answer 9 · answered Apr 01 '09 at 10:04

0

For small files you may need to had this lines in order to avoid crazy percentages:

sys.stdout.write("\r%2d%%" % percent)

sys.stdout.flush()

Cheers

answered Apr 01 '09 at 10:04

score 0 · Answer 10 · edited Jan 22 '22 at 00:37

0

To avoid progress values like 106% or similar, follow the below logic

percent = min(int(count * blockSize * 100 / totalSize), 100)

edited Jan 22 '22 at 00:37

Flair

2,609
1
29
41

answered Jan 18 '22 at 05:23

Ashutosh Kumbhar

39
4

How to write a download progress indicator in Python?

10 Answers10

Linked