4

I've been using this code to mount Colab with Google Drive and download any file by pasting in download URL but I have noticed that it take quite long even though the files are few megabytes in size. Is there anything that can be done to improve it.

**First cell:**
from google.colab import drive
drive.mount('/content/gdrive')
root_path = 'gdrive/My Drive/' 

**Second cell:**
import requests  
file_url = "DOWNLOAD URL HERE"

r = requests.get(file_url, stream = True)  

with open("/content/gdrive/My Drive/FILE NAME HERE", "wb") as file:  
    for block in r.iter_content(chunk_size = 1024): 
         if block:  
             file.write(block)
bad_coder
  • 11,289
  • 20
  • 44
  • 72
M.Gharabeel
  • 43
  • 1
  • 1
  • 4

2 Answers2

11

I prefer using !wget method.

!wget "Specify URL you want to download" -P "specify DIRECTORY where you want contents of URL"

For example.

!wget "https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_Dataset.zip" -P "/content/drive/My Drive/imgcaptiongen/data"

It's much easier this way.

Pavindu
  • 2,684
  • 6
  • 44
  • 77
Abhijeet Konduskar
  • 111
  • 3
0

Don't download the file into Google Drive, as there's overhead to access Google Drive in Colab, and especially to write files there.

If this file is temporary, just download it to /tmp (or use tempfile.gettempdir to make the code prettier). If it's not temporary, still consider downloading it to a temporary folder and then copying it at the end of the download to Drive (while continuing to work with the local copy for speed).

Barak Itkin
  • 4,872
  • 1
  • 22
  • 29
  • Can you please help me integrate it, I've tried but I didn't get tangible outcomes. – M.Gharabeel Jun 10 '20 at 08:26
  • @M.Gharabeel - Download a file with requests using https://stackoverflow.com/a/39217788/748102, but modify the created file name/path to be `tempfile.mkstemp()` (see https://docs.python.org/3/library/tempfile.html#tempfile.mkstemp). Finally, move the file to Drive with `shutil.move` (https://docs.python.org/3/library/shutil.html#shutil.move) specifying `copy_function=shutil.copy` (see the documentation for why) – Barak Itkin Jun 10 '20 at 08:41