8

i have problem with streaming download large file (about 1.5 GB) in python-requests v. 2.0.1

with open("saved.rar",'wb') as file:
    r = session.get(url,stream=True,timeout=3600)
    for chunk in r.iter_content(chunk_size=1024):
        if chunk:
            file.write(chunk)
            file.flush()

I tested it few times on my vps and sometimes it downloaded 200mb, 500mb or 800mb and saved it without any error. It doesnt reached the timeout, just stopped like finish downloading.

Host where im downloading this file is stable because i dont have any problems to download this file in browser.

There is any way to be download large file in python-requests and be 100% sure its whole file?

@Edit

I've solved it using urllib, problem is only with requests. anyway thanks for help.

  • 5
    I am distracted by the photo in your avatar!lol! I am kidding. Not so common in SO.... Check out this link...http://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py – Jack_of_All_Trades Nov 01 '13 at 12:31
  • @Jack_of_All_Trades Unfortunately, I did not find anything there. This is the same code that I have. –  Nov 01 '13 at 13:15

1 Answers1

0

There might be several issues that will cause download to be interrupted. Network issues, etc. But we know the file size before we start the download to check if you have downloaded the whole file, you can do this using urllib:

site = urllib.urlopen("http://python.org")
meta = site.info()
print meta.getheaders("Content-Length")

Using requests:

r = requests.get("http://python.org")
r.headers["Content-Length"]
adam
  • 238
  • 4
  • 14
  • 1
    Okey but i'm wondering what is wrong with requests because i needed to download file 10 times to get whole file, when i dont have same problem with curl. –  Nov 01 '13 at 13:18
  • You might need os.fsync() as explained in http://stackoverflow.com/a/7127162/2938650. Try to log any errors and exceptions. I suspect there might be out of memory message. – adam Nov 01 '13 at 14:53