3

I'm working on a project that requires multiple files to be uploaded to Google Drive via the API. The number of files can be in the dozens and their size can be in multiple Megabytes.

My question is this: Is it a better strategy to send the files one-by-one to Google Drive, or should I send a bunch of files in parallel? Does Google Drive parallelize the requests or just perform them in sequence one-by-one? Is there any documentation that describes this behavior?

Thanks in advance!

DataGrump
  • 213
  • 3
  • 12
  • 1
    http://stackoverflow.com/questions/10311969/what-is-the-limit-on-google-drive-api-usage I would say, you can upload them in parralel, it will simply get deducted from your quota. But i'm not an expert, so hence the comment. – Tschallacka Dec 08 '16 at 12:34
  • 1
    @Tschallacka Thanks, but my question pertains primarily to the amount of time it would take to complete. I assume the quota is handled the same. – DataGrump Dec 08 '16 at 12:37
  • 1
    well, there is only one way to find out, and that's by simply doing it in parralel and sequential and measure the times. I suggest you use webworkers to get true parralelism. – Tschallacka Dec 08 '16 at 12:43
  • 1
    @Tschallacka You are right, and I've done that but I'm getting inconsistent results (perhaps due to my own internet connection) so I'm trying to get a somewhat official answer. Generally speaking it seems that SOMETIMES, for a low number of files, it's good to go parallel. – DataGrump Dec 08 '16 at 13:01
  • http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser might also help – Tschallacka Dec 08 '16 at 13:11

2 Answers2

4

OK, so it seems to me that regardless of whether you send the files one-by-one or in bulk, they are in practicality handled one-by-one.

I've conducted a test in which I uploaded 1 file, 2 files, 3 files, 4 files, 5 files and 10 files in parallel and recorded them time it took. I did this 9 times for each number of files and recorded the median time in seconds. All the files are in fact the same file (approximately 3 MBs in size):

Here are the results:

1 file: 9.6 secs total - 9.6 secs per file

2 files: 19.7 secs total - 9.8 secs per file

3 files: 26.5 secs total - 8.8 secs per file

4 files: 34.1 secs total - 8.5 secs per file

5 files: 44.6 secs total - 8.9 secs per file

10 files: 84.1 secs total - 8.41 secs per file

The bottom line is that regardless of how many upload requests you send Google Drive (assuming you don't get a 403), the amount of time it would take per file is the same. In my case, 8-9 seconds per file.

I guess the upload requests are queued on Google's end.

DataGrump
  • 213
  • 3
  • 12
0

There is no one "best strategy". It depends on what you want to achieve in your application. Google Drive can handle serial or parallel uploads, but bear in mind that there are undocumented limits on how rapidly you can send multiple requsts. If you send too rapidly, you'll encounter "304 rate limit" errors, which is Google's way of saying "slow down or we'll treat this as a DoS attack".

pinoyyid
  • 21,499
  • 14
  • 64
  • 115