2

I'm working on a Python application which is supposed to log different types of entries. The log file's format is JSON Lines http://jsonlines.org/. If the script detects a Wi-Fi connection, it updates the file, uploading it using PyDrive (all the authentication process omitted here, but fully functional).

SetContentFile(filename)
Upload()

The script runs on Pythonista on my iPhone SE. I installed PyDrive through pip which I got thanks to StaSh https://github.com/ywangd/stash. Right now the file I'm uploading/updating is not big as it only has a couple of lines, however it came to my mind that in the future, when this log file's size is 2GB (to say a number) the uploading/updating process will take too much time.

The application does nothing but append lines to the file.

Is it possible to just append the new line(s) to the file stored on Google's servers instead of uploading/updating the whole file? Do I need to fetch some metadata and from the files (local and Drive) and specify it as a parameter for some function? (SetContentFile() doesn't seem to take extra parameters)

I didn't find a function named something similar to append() in the docs https://pythonhosted.org/PyDrive/filemanagement.html.

I did find, however, the following claim which I don't fully understand:

Upload(param=None) 
   **Upload/update file by choosing the most efficient method.**
   Parameters:  param (dict.) – additional parameter to upload file.
   Raises:  ApiRequestError

https://pythonhosted.org/PyDrive/pydrive.html#pydrive.files.GoogleDriveFile.SetContentFile

Does the "most efficient method" mean that only the last lines appended to the local copy of the file will be uploaded? Or does it refer to something such as the protocol used?

Maganna Dev
  • 189
  • 1
  • 11
  • Most efficient method is referring to selecting the medial, multipart or resumable upload type. The resumable upload type might avoid uploading chunks already on the server, but it's not clear that it does this at all and I wouldn't count on it. – Simon Hibbs Jan 22 '19 at 12:07
  • Yea append would be great, for your application, you could upload seperate files for each 'change' and have a database to search and organize the files. I do wonder though if appending to a file is less TCP packets than, starting a new upload and then closing it for every bit of data. If it is indeed less packets, then append would be an excellent improvement, for people uploading lots of data at simultanaously. /the less tcp packets the better? For example, if i'm uploading 1,000 files to the server consecutively, i'd rather append them all to one big file .. – Edo Edo Aug 02 '19 at 05:41

0 Answers0