0

I am trying to write a program that will connect to our Google apps domain and change the ownership of all files owned by a specific user. I have the code working to some degree but when I try to send a batch http request with all of my insert permission requests, a large percentage of them return with the following.

<HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B6zuNvhcekkVNzZ0M3RFdTA1MXc/permissions/18218617888593763709 returned "Internal Error">

It's trivial to write code to retry the failed files but I would like to know why this is happening and if there is anyway I can prevent this behavior, thereby increasing the speed of my program.

Below is my code to change the permissions. I do realize there is a maximum number of requests per batch operation but in our dev environment I'm controlling the number of files that are located on the google drive account, so I'm no where near this number. 

def insertNewOwnerPermissions(self, files, new_owner):
    """Insert a new owner into the file

    Args:
        files: list of files that we want to process 
        new_owner: new owner's email address
    """

    new_permission = {
        'value': new_owner,
        'type': 'user',
        'role': 'owner'
    }

    #Add all permission requests to batch
    for file in files:
        self.batch.add(self.driveService.permissions().insert(fileId=str(file['id']), body=new_permission), callback=self.insertCallback, request_id=str(file['id']))

    #Send the batch request
    self.batch.execute(http=self.http)



def insertCallback(self, request_id, response, exception):
    print "id:"+str(request_id)
    print "response:"+str(response)
    print "exception:"+str(exception)
LexNix
  • 377
  • 1
  • 3
  • 10

1 Answers1

0

It might be a rate limit problem. When you send a batch, it gets as far as a front end server, which unpacks the batch and then fires the items at Drive. Drive can then barf at the update rate. When I've hit this before, I've got a 403 Rate Limit, rather than a 500, but Drive SDK errors are notoriously inconsistent.

I ended up giving up on batch requests and reverting back to individually timed requests. This proved far more efficient than uploading a batch and then following up with large numbers of retries.

pinoyyid
  • 21,499
  • 14
  • 64
  • 115
  • I thought about going back to individual requests but we have users with thousands of files in their drive accounts. I'm afraid if I do this, the processing time will be outrageous. Any suggestions on ways I can increase the efficiency of these operations? – LexNix Mar 18 '14 at 18:15
  • In order to reduce the elapsed time for a batch of requests, you need to submit them below the rate limit. Drive uses a token/bucket system. From my testing the bucket holds around 30 tokens and the replenish rate is around 1 every 2 seconds. The exact numbers are a secret. Once you exceed this rate, you are in retry hell so your elapsed time sucks. My testing is 6 months old, http://stackoverflow.com/questions/18578768/403-rate-limit-on-insert-sometimes-succeeds so things might have changed. That's a secret too. – pinoyyid Mar 20 '14 at 05:23