Python: download files from google drive using url

Question

I am trying to download files from google drive and all I have is the drive's URL.

I have read about google API that talks about some drive_service and MedioIO, which also requires some credentials( mainly JSON file/OAuth). But I am unable to get any idea about how it is working.

Also, tried urllib2.urlretrieve, but my case is to get files from the drive. Tried wget too but no use.

Tried PyDrive library. It has good upload functions to drive but no download options.

Any help will be appreciated. Thanks.

Answer 1

If by "drive's url" you mean the shareable link of a file on Google Drive, then the following might help:

import sys
import requests


def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download&confirm=1"

    session = requests.Session()

    response = session.get(URL, params={"id": id}, stream=True)
    token = get_confirm_token(response)

    if token:
        params = {"id": id, "confirm": token}
        response = session.get(URL, params=params, stream=True)

    save_response_content(response, destination)


def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith("download_warning"):
            return value

    return None


def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk:  # filter out keep-alive new chunks
                f.write(chunk)


def main():
    if len(sys.argv) >= 3:
        file_id = sys.argv[1]
        destination = sys.argv[2]
    else:
        file_id = "TAKE_ID_FROM_SHAREABLE_LINK"
        destination = "DESTINATION_FILE_ON_YOUR_DISK"
    print(f"dowload {file_id} to {destination}")
    download_file_from_google_drive(file_id, destination)


if __name__ == "__main__":
    main()

The snipped does not use pydrive, nor the Google Drive SDK, though. It uses the requests module (which is, somehow, an alternative to urllib2).

When downloading large files from Google Drive, a single GET request is not sufficient. A second one is needed - see wget/curl large file from google drive.

Answer 2

I recommend gdown package.

pip install gdown

Take your share link

https://drive.google.com/file/d/0B9P1L--7Wd2vNm9zMTJWOGxobkU/view?usp=sharing

and grab the id - eg. 1TLNdIufzwesDbyr_nVTR7Zrx9oRHLM_N by pressing the download button (look for at the link), and swap it in after the id below.

import gdown

url = 'https://drive.google.com/uc?id=0B9P1L--7Wd2vNm9zMTJWOGxobkU'
output = '20150428_collected_images.tgz'
gdown.download(url, output, quiet=False)

Answer 3

Having had similar needs many times, I made an extra simple class GoogleDriveDownloader starting on the snippet from @user115202 above. You can find the source code here.

You can also install it through pip:

pip install googledrivedownloader

Then usage is as simple as:

from google_drive_downloader import GoogleDriveDownloader as gdd

gdd.download_file_from_google_drive(file_id='1iytA1n2z4go3uVCwE__vIKouTKyIDjEq',
                                    dest_path='./data/mnist.zip',
                                    unzip=True)

This snippet will download an archive shared in Google Drive. In this case 1iytA1n2z4go3uVCwE__vIKouTKyIDjEq is the id of the sharable link got from Google Drive.

Answer 4

Here's an easy way to do it with no third-party libraries and a service account.

pip install google-api-core and google-api-python-client

from googleapiclient.discovery import build
from googleapiclient.http import MediaIoBaseDownload
from google.oauth2 import service_account
import io

credz = {} #put json credentials her from service account or the like
# More info: https://cloud.google.com/docs/authentication

credentials = service_account.Credentials.from_service_account_info(credz)
drive_service = build('drive', 'v3', credentials=credentials)

file_id = '0BwwA4oUTeiV1UVNwOHItT0xfa2M'
request = drive_service.files().get_media(fileId=file_id)
#fh = io.BytesIO() # this can be used to keep in memory
fh = io.FileIO('file.tar.gz', 'wb') # this can be used to write to disk
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
    status, done = downloader.next_chunk()
    print("Download %d%%." % int(status.progress() * 100))

Answer 5

PyDrive allows you to download a file with the function GetContentFile(). You can find the function's documentation here.

See example below:

# Initialize GoogleDriveFile instance with file id.
file_obj = drive.CreateFile({'id': '<your file ID here>'})
file_obj.GetContentFile('cats.png') # Download file as 'cats.png'.

This code assumes that you have an authenticated drive object, the docs on this can be found here and here.

In the general case this is done like so:

from pydrive.auth import GoogleAuth

gauth = GoogleAuth()
# Create local webserver which automatically handles authentication.
gauth.LocalWebserverAuth()

# Create GoogleDrive instance with authenticated GoogleAuth instance.
drive = GoogleDrive(gauth)

Info on silent authentication on a server can be found here and involves writing a settings.yaml (example: here) in which you save the authentication details.

Answer 6

There's in the docs a function that downloads a file when we provide an ID of the file to download,

from __future__ import print_function

import io

import google.auth
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from googleapiclient.http import MediaIoBaseDownload


def download_file(real_file_id):
    """Downloads a file
    Args:
        real_file_id: ID of the file to download
    Returns : IO object with location.

    Load pre-authorized user credentials from the environment.
    TODO(developer) - See https://developers.google.com/identity
    for guides on implementing OAuth2 for the application.
    """
    creds, _ = google.auth.default()

    try:
        # create drive api client
        service = build('drive', 'v3', credentials=creds)

        file_id = real_file_id

        # pylint: disable=maybe-no-member
        request = service.files().get_media(fileId=file_id)
        file = io.BytesIO()
        downloader = MediaIoBaseDownload(file, request)
        done = False
        while done is False:
            status, done = downloader.next_chunk()
            print(F'Download {int(status.progress() * 100)}.')

    except HttpError as error:
        print(F'An error occurred: {error}')
        file = None

    return file.getvalue()


if __name__ == '__main__':
    download_file(real_file_id='1KuPmvGq8yoYgbfW74OENMCB5H0n_2Jm9')

---

This bears the question:

How do we get the file ID to download the file?

Generally speaking, a URL from a shared file from Google Drive looks like this

https://drive.google.com/file/d/1HV6vf8pB-EYnjcJcH65eGZVMa2v2tcMh/view?usp=sharing

where 1HV6vf8pB-EYnjcJcH65eGZVMa2v2tcMh corresponds to fileID.

You can simply copy it from the URL or, if you prefer, it's also possible to create a function to get the fileID from the URL.

For instance, given the following url = https://drive.google.com/file/d/1HV6vf8pB-EYnjcJcH65eGZVMa2v2tcMh/view?usp=sharing,

def url_to_id(url):
    x = url.split("/")
    return x[5]

Printing x will give

['https:', '', 'drive.google.com', 'file', 'd', '1HV6vf8pB-EYnjcJcH65eGZVMa2v2tcMh', 'view?usp=sharing']

And so, as we want to return the 6th array value, we use x[5].

Answer 7

import requests

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id , 'confirm': 1 }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)    

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

if __name__ == "__main__":
    file_id = 'TAKE ID FROM SHAREABLE LINK'
    destination = 'DESTINATION FILE ON YOUR DISK'
    download_file_from_google_drive(file_id, destination)

Just repeating the accepted answer but adding confirm=1 parameter so it always downloads even if the file is too big

Answer 8

This has also been described above,

   from pydrive.auth import GoogleAuth
   gauth = GoogleAuth()
   gauth.LocalWebserverAuth()
   drive = GoogleDrive(gauth)

This creates its own server too do the dirty work of authenticating

   file_obj = drive.CreateFile({'id': '<Put the file ID here>'})
   file_obj.GetContentFile('Demo.txt') 

This downloads the file

Answer 9

# Importing [PyDrive][1] OAuth
from pydrive.auth import GoogleAuth

def download_tracking_file_by_id(file_id, download_dir):
    gauth = GoogleAuth(settings_file='../settings.yaml')
    # Try to load saved client credentials
    gauth.LoadCredentialsFile("../credentials.json")
    if gauth.credentials is None:
        # Authenticate if they're not there
        gauth.LocalWebserverAuth()
    elif gauth.access_token_expired:
        # Refresh them if expired
        gauth.Refresh()
    else:
        # Initialize the saved creds
        gauth.Authorize()
    # Save the current credentials to a file
    gauth.SaveCredentialsFile("../credentials.json")

    drive = GoogleDrive(gauth)

    logger.debug("Trying to download file_id " + str(file_id))
    file6 = drive.CreateFile({'id': file_id})
    file6.GetContentFile(download_dir+'mapmob.zip')
    zipfile.ZipFile(download_dir + 'test.zip').extractall(UNZIP_DIR)
    tracking_data_location = download_dir + 'test.json'
    return tracking_data_location

The above function downloads the file given the file_id to a specified downloads folder. Now the question remains, how to get the file_id? Simply split the url by id= to get the file_id.

file_id = url.split("id=")[1]

Answer 10

I tried using google Colaboratory: https://colab.research.google.com/

Suppose your sharable link is https://docs.google.com/spreadsheets/d/12hiI0NK7M0KEfscMfyBaLT9gxcZMleeu/edit?usp=sharing&ouid=102608702203033509854&rtpof=true&sd=true

all you need is id that is 12hiI0NK7M0KEfscMfyBaLT9gxcZMleeu

command in cell

!gdown 12hiI0NK7M0KEfscMfyBaLT9gxcZMleeu

run the cell and you will see that file is downloaded in /content/Amazon_Reviews.xlsx

Note: one should know how to use Google colab

Answer 11

This example is based on an similar to RayB, but keeps the file in memory and is a little simpler, and you can paste it into colab and it works.

import googleapiclient.discovery
import oauth2client.client
from google.colab import auth
auth.authenticate_user()

def download_gdrive(id):
  creds = oauth2client.client.GoogleCredentials.get_application_default()
  service = googleapiclient.discovery.build('drive', 'v3', credentials=creds)
  return service.files().get_media(fileId=id).execute()

a = download_gdrive("1F-yaQB8fdsfsdafm2l8WFjhEiYSHZrCcr")

Answer 12

I used the accepted solution for a long period, but now google has changed the download warning response so it does not work anymore.

I am now using the API as it is a safer way to ensure it won't suddenly stop, but I could also make it work parsing the response HTML looking for the download url as following:

import requests
from html.parser import HTMLParser

class MyHTMLParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.action = None

    def handle_starttag(self, tag, attrs):
        if tag == "form":
            for name, value in attrs:
                if name == "id" and value == "download-form":
                    for name, value in attrs:
                        if name == "action":
                            self.action = value

DOWNLOAD_URL = 'https://docs.google.com/uc?export=download'
session = requests.Session()
response = session.get(file_url, params={'id': id}, stream=True)

content_type = response.headers['content-type']
if content_type == 'text/html; charset=utf-8':
    parser = MyHTMLParser()
    parser.feed(response.text)
    download_url = parser.action
    response = session.post(download_url, stream=True)
    
file = response.content

Answer 13

Fixed version for 2023 + generator for tracking progress

import requests


def download_file_from_google_drive(file_id, destination, chunk_size=32768):
    url = "https://docs.google.com/uc?export=download"

    session = requests.Session()
    params = {'id': file_id, 'confirm': 1}
    response = session.get(url, params=params, stream=True)

    for i, chunk_size_ in save_response_content(response, destination, chunk_size):
        yield i, chunk_size_


def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None


def save_response_content(response, destination, chunk_size):
    with open(destination, "wb") as f:
        for i, chunk in enumerate(response.iter_content(chunk_size)):
            if chunk:  # filter out keep-alive new chunks
                f.write(chunk)
                yield i, chunk_size


if __name__ == '__main__':
    file_id = '...'
    destination = '...'
    for i, chunk_size in download_file_from_google_drive(file_id, destination):
        print(i, chunk_size)

Answer 14

For those who are interested in the link to download via HTTP, the Google API and most clients provide a webContentLink field containing it (note the file permissions to use it)

Answer 15

You can install https://pypi.org/project/googleDriveFileDownloader/

pip install googleDriveFileDownloader

And download the file, here is the sample code to download

from googleDriveFileDownloader import googleDriveFileDownloader
a = googleDriveFileDownloader()
a.downloadFile("https://drive.google.com/uc?id=1O4x8rwGJAh8gRo8sjm0kuKFf6vCEm93G&export=download")