1

I'm using PyDrive to upload files from my RPi to a specific folder in my Google Drive. It is successfully working, but the speed is terribly slow. For a .npy file (binary numpy file) that is only 40kB, the upload speed is around 2 seconds. When I try uploading a different file (.pptx) that is 2MB, the upload speed is around 5 seconds. I also tried this on my Mac, and it has the same upload speed.

Is there a better way to do this? I need an upload speed that is less than a second since I'm collecting data every second. Here is the code:

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import os
import time

credentials = '/***/pydrive_credentials.txt'
directory = '/***/remote_dir'
gauth = GoogleAuth()
gauth.LoadCredentialsFile(credentials)
# gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)


# get id of designated folder in Google Drive 
folder = drive.ListFile({'q': "title = 'sample pydrive folder' and trashed=false"}).GetList()[0]

for filename in os.listdir(directory):

    f = drive.CreateFile({'title': filename, 'parents': [{'id': folder['id']}]})
    # f = drive.CreateFile()
    filepath = os.path.join(directory, filename)
    f.SetContentFile(filepath)
    start = time.time()
    f.Upload()
    end = time.time()
    print(end-start)

    # delete file after upload
    # os.remove(filepath)
    
    # to ensure no memory leakage
    f = None
    filepath = None

    print("Uploaded: {}".format(filename))
JacksonPro
  • 3,135
  • 2
  • 6
  • 29
Joseph
  • 11
  • 2
  • Hard to say, could it be an internet connection quality thing? Have you tried on a different network? – Gino Mempin Jan 22 '21 at 03:44
  • 1
    The PyDrive API may provide a batch mode that allows you to upload multiple files in a single request. If you are routinely uploading multiple files at the same time, utilizing such a mode should speed things up quite a bit, especially if the files are small. - It is strange that uploading a single small file takes anything like 2 seconds. – CryptoFool Jan 22 '21 at 03:47
  • I did try on different networks and still the same result. My upload speed is around 100mpbs so it shouldn't be an issue. @Steve I don't see any batch mode in PyDrive API. It seems there is no other way around but to upload them one by one based on the following links [1](https://stackoverflow.com/questions/55972048/how-to-upload-multiple-files-upload-in-google-drive) [2](https://stackoverflow.com/questions/41039152/best-strategy-to-upload-multiple-files-to-google-drive-via-javascript-api)? – Joseph Jan 22 '21 at 04:11

0 Answers0