1

I'm working on a node script to download all the images that are shared with my account on Google Drive.

I'm hitting a wall just as others have posted about on Stack Overflow: Google Drive API file watch rate limits.

The exponential backoff makes a lot of sense to me, I just don't know how to go about implementing that.

Any insight I could get into this would be incredibly helpful. Even just a "get started by..." would be great!

I've added the method I'm working on below. Auth and everything is working just fine, it's just a matter of receiving the userRateLimitExceeded error.

Any and all help would be fantastic and greatly appreciated.

Thank you!

/**
 * Download all of the shared images.
 *
 * @param {google.auth.OAuth2} auth An authorized OAuth2 client.
 */
function downloadImages(auth) {
  const gDrive = google.drive({
    version: 'v3',
    auth: auth
  });

  gDrive.files.list({
    q: 'sharedWithMe = true and mimeType = "image/jpeg"'
  }, (err, resp) => {
    if(err) {
      console.log('The API returned an error: ' + err);
      return;
    }

    if(!resp.files.length) {
      console.error('No files found.');
    } else {
      // Remove existing images.
      // removeImages();

      _.each(resp.files, (file) => {
        if(fs.existsSync(IMAGE_DIR + file.name)) {
          return;
        }

        gDrive.files.get({
          fileId: file.id
        })
        .on('end', () => {
          console.log(chalk.green(file.name + ' successfully downloaded.'));
        })
        .on('error', (err) => {
          console.log(err);
        })
        .pipe(fs.createWriteStream(IMAGE_DIR + file.name));
      });
    }
  });
}

EDIT: I looked into batching, but I guess google-api-nodejs-client doesn't support batches. I tried a third-party lib called "Batchelor". Still can't get it to work for the life of me. :(

Community
  • 1
  • 1
Will
  • 914
  • 3
  • 11
  • 19
  • When you are getting rate limited, what does `gDrive.fkiles.get()` do? Do you get a specific error in the `.on('error', ...)` event handler that you can detect? If so, what specific error is that? Basically, if you can help us know how to detect when you're getting rate limited, we can help with the code to deal with that. – jfriend00 Dec 18 '16 at 01:34
  • @jfriend00 - It didn't look like it even got to the error handler. It would just return with an error object saying "userRateLimitExceeded" or something. And I say it didn't get to the error handler because if I did something like `console.error('error', error)`, I wouldn't see my log label at all. :( – Will Dec 18 '16 at 19:47

3 Answers3

2

It's not simple, especially from an aync language like JavaScript. Firstly, do NOT do exponential backoff, as I explained in my answer to the question you cited. You will end up with a massive delay between API calls.

You can have a poke around the code in https://github.com/pinoyyid/ngDrive/blob/master/src/http_s.ts This is an Angular 1 service that handles GDrives idiosyncrasies, including 403. It doees it by putting requests into an internal queue, and I then have a process that takes items off the queue at a variable rate to maximise throughput, but minimise the 403's and attendant retries.

Batching makes it even worse, so don't go down that road. It's not the rate of http requests which is the limiting factor, it's the rate of requeststs to the GDrive internal systems. A batch of requests gets fired into GDrive in rapid succession, os is more likely to trigger a 403.

pinoyyid
  • 21,499
  • 14
  • 64
  • 115
  • Appreciate the reply. I wasn't able to get the simple example of ngDrive working locally, unfortunately. Something about "401. That’s an error. Error: invalid_client no registered origin". I think this is where I give up on this idea. :/ – Will Dec 18 '16 at 20:24
  • that 401 is just config stuff. You need to 1/ create a clinet ID in the API console, 2/ configure that into your app, 3/ serve your app from one of the authorised URLs that you specified in step 1 – pinoyyid Dec 19 '16 at 09:01
1

Based on this documentation, the error 403 or the userRateLimitExceeded error that you got means that the per-user limit from the Developer Console has been reached.

Just take Note that the Drive API has a:

  • Courtesy daily quota 1,000,000,000 requests/day
  • Courtesy daily quota per second 100 requests/second
  • Per User Limit 10 requests/second/user

Here is the screenshot of the default quota of Drive API

enter image description here

So the suggested action for your error are:

  • Raise the per-user quota in the Developer Console project. (Use this link to ask for more quota)

  • If one user is making a lot of requests on behalf of many users of a Google Apps domain, consider a Service Account with authority delegation (setting the quotaUser parameter).

  • Use exponential backoff.

For more information, check these SO questions:

Community
  • 1
  • 1
KENdi
  • 7,576
  • 2
  • 16
  • 31
  • "Per User Limit 10 requests/second/user" is incorrect. The rate throttling is more complex than that, and isn't publicly documented. – pinoyyid Dec 18 '16 at 17:01
  • Check your Developer Console, you will see that User - 100s has a default quota of 1,000. So it means that you have a Per User Limit 10 requests/second/user. Check the screenshot that I provide. – KENdi Dec 19 '16 at 00:37
  • It doesn't mean that at all. There are many different limits. The most common one that people trip over is the rate throttling which works on a token/bucket algorithm which isn't published. – pinoyyid Dec 19 '16 at 09:00
1

With the rate limits, the one that you have to watch out for is the per user per second.

I used a script that sends a batch of 10 requests every 1.3 seconds, that scripts works fine.

Sending a batch of 100 requests over 13 seconds however starts getting 403 errors partway through.

So the conclusion: watch out for the per second, not per 100 second limit.

Dijkgraaf
  • 11,049
  • 17
  • 42
  • 54
Hilmi Hamdan
  • 21
  • 2