0

I am trying to upload multiple files to Google drive using the PHP SDK. For this I am calling the function below iteratively passing the required parameters:

function insertFile($driveService, $title, $description, $parentId, $fileUrl) {
    global $header;

    $file = new Google_DriveFile();
    $file->setTitle($title);
    $file->setDescription($description);
    $mimeType= "application/vnd.google-apps.folder";

    if ($fileUrl != null) {
      $fileUrl = replaceSpaceWithHtmlCode($fileUrl);
      $header = getUrlHeader($fileUrl);
      $mimeType = $header['content-type'];
    }
    $file->setMimeType($mimeType);
    $parent = new Google_ParentReference();
    // Set the parent folder.
    if ($parentId != null) {
      $parent->setId($parentId);
      $file->setParents(array($parent));
    }

    try {
      $data = null;
      if ($fileUrl != null) {
        if (hasErrors($driveService, $fileUrl) == True) {
          return null;
        }
        $data = file_get_contents($fileUrl);
    }

    $createdFile = $driveService->files->insert($file, array(
      'data' => $data,
      'mimeType' => $mimeType,
    ));

      return $createdFile;
    } catch (Exception $e) {
        echo "Error: 12";
    return null;
    }
}

I am running this app on the Google App Engine.

However, I am unable to upload all the files I pass to it. For example, if I pass about 12-15 files, only 10-11 get uploaded, and sometimes all get uploaded, even though all parameters are correct. I have caught the exception when it fails to create a file and this says it is unable to create a file, for the files that are not uploaded. I don't see any warnings or errors in the logs on the app engine.

Am I missing something? Can someone please point me where I should be looking to correct this and make it reliable enough to upload all files given to it?

The HTTP response that I get when I try to upload 30 files is this:

PHP Fatal error:  The request was aborted because it exceeded the maximum execution time
jobin
  • 2,600
  • 7
  • 32
  • 59

1 Answers1

0

Check the http response to see the detailed reason. It might be that you are hitting the throttle limit and getting a 403 rate limit response.

pinoyyid
  • 21,499
  • 14
  • 64
  • 115
  • Nothing in the logs seem to say that the throttle limit has been reached and if the problem would have been the throttle limit, I am not sure if it would have been the case that the last few files are not uploaded everytime, 4 upload when 5 are given and 13 upload when 15 are given. – jobin Nov 19 '13 at 07:51
  • The throttling works using a token bucket. A burst of inserts works fine until the tokens are exhausted, around 20 in my experience. After that, the tokens seem to be replenished at a rate of around one every 2 seconds. Any insert attempts with an empty bucket will cause a 403 exception. Check the http response. – pinoyyid Nov 19 '13 at 08:16
  • That looks like an http timeout. Can you change the timeout? I would also suggest (1) look at the actual http traffic, don't just rely on logs, (2) try slowing down your insert rate to one every 3 seconds to see if that fixes it. See http://stackoverflow.com/questions/18529524/403-rate-limit-after-only-1-insert-per-second – pinoyyid Nov 19 '13 at 09:45
  • Can you please tell me how do I change the timeout? I have tried 1 per second, but that made no difference, it was still unreliable. – jobin Nov 19 '13 at 10:01
  • One per second is too fast. Try one per 3 seconds, at least to determine if that is the cause. I don't know PHP - sorry. I suggest ask a new question "how to set the timeout of urlfetch on appengine php". The docs say the default is 5s which is way to low. – pinoyyid Nov 19 '13 at 15:23