Upload large file not working - Google Drive Python API

Question

This script works for small files, but not when I try to upload a large file (250MB). When I manually upload the same large file to GD it takes less than 10 seconds, so I assume my connection is not the problem.

upload.py

from __future__ import print_function
import os
import sys

from apiclient.http import MediaFileUpload
from apiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools

try:
    import argparse
    flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
    flags = None

SCOPES = 'https://www.googleapis.com/auth/drive.file'
store = file.Storage(r'C:\Users\lucas.rezende\.credentials\storage.json')
creds = store.get()

if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets(r'C:\Users\lucas.rezende\.credentials\client_secret.json', scope=SCOPES)
    creds = tools.run_flow(flow, store, flags) if flags else tools.run(flow, store)
DRIVE = build('drive', 'v3', http=creds.authorize(Http()))

FILES = (
    ('OfertasMensais_20170418_n.xlsx', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'),
)

for filename, mimeType in FILES:

    media_body = MediaFileUpload(filename, chunksize=1024*256, resumable = True)

    folder_id = '0000'
    metadata = {'name': filename, 'parents': [folder_id]}

    if mimeType:
        metadata['mimeType'] = mimeType
    res = DRIVE.files().create(body=metadata, media_body=filename).execute()

    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

When I run python uploadfile.py cmd screen stays like that eternally:

Can someone help to discover how to make this work? I am not a professional programmer and I am stuck into this for almost two hours trying to make this work.

Where did you get that MIME type? http://stackoverflow.com/a/36957953/130453 — corn3lius, Apr 20 '17 at 18:36
@corn3lius I got from the 3rd answer at this post: http://stackoverflow.com/questions/11894772/google-drive-mime-types-listing, but actually I want no conversion. Just upload my xlsx file to GD. — Lucas Rezende, Apr 20 '17 at 18:40

corn3lius · Answer 1 · 2017-04-20T20:01:59.233

1

Following the chunked paradigm, you need to specifically call the next_chunk() to continue with an upload. see here: https://developers.google.com/api-client-library/python/guide/media_upload#resumable-media-chunked-upload

for filename, mimeType in FILES:
    media_body = MediaFileUpload(filename, chunksize=1024*256, resumable = True) 

    if mimeType:
        metadata['mimeType'] = mimeType

    req = DRIVE.files().insert(body=metadata, media_body=filename)
    res = None
    while res is None:
        status, res = req.next_chunk()
        if status :
            print('Uploading %d%% "%s" (%s)' % (status.progress(), filename, res['mimeType']))
    print("Upload Complete!")

edited Apr 20 '17 at 20:01

answered Apr 20 '17 at 19:25

corn3lius

4,857
2
31
36

I updated my code to this one but it seems to be missing some import. I got an `NameError: name 'request' is not ndefined`. I am currently for it. – Lucas Rezende Apr 20 '17 at 19:35
From what I read on the link you sent I could also remove the `chunksize=1024*256` from the `MediaFileUpload` for a Single Request... – Lucas Rezende Apr 20 '17 at 19:37
@LucasRezende Just edittied it `req` is `request`. You could do that too! you would need to remove the `resumeable` flag as well. – corn3lius Apr 20 '17 at 20:03
I see that you posted the `req` using `insert` method, but in API v3 this method is not used anymore. It uses `create().execute()` and when I put it that way I get an error saying that req has no `next_chunk()` method. Any suggestion? – Lucas Rezende Apr 20 '17 at 20:45
my 2c is that when any library starts getting in the way, stop using it. Use the REST API directly, it's well formed and easy to use. That way you also get to use a combination of v2 and v3 while the shortfalls in v3 are addressed. – pinoyyid Apr 21 '17 at 07:48

Cameron Roberts · Answer 2 · 2020-02-20T16:11:13.630

1

The solution for this with v3 is to use the chunked approach, but with the create() function rather than insert()

            res = None
            media_body = MediaFileUpload(filename, chunksize=1024*256, resumable = True)

            drive_request = self.drive.files().create(body=metadata,media_body=media_body)
            while res is None:
                status, res = drive_request.next_chunk()

edited Feb 20 '20 at 16:11

answered Nov 25 '19 at 22:26

Cameron Roberts

7,127
1
20
32

I think the sintax is wrong. You should use DRIVE instead of drive_request to call the next_chunk() method, right? – Eduardo Fernandes Feb 20 '20 at 12:38
1

@EduardoFernandes yes you are correct, it seems like I edited the code after pasting it to match what OP had, but I missed changing it in both places. I have switched it back to match the code I have in production instead. – Cameron Roberts Feb 20 '20 at 16:31

Upload large file not working - Google Drive Python API

2 Answers2

Linked