Memory allocation error on basic Google Drive PHP API upload script

Question

[EDIT] As the error specifies, my php.ini allocates 1GB of RAM per script (memory_limit = 1G), which should be more than enough, so I'm not asking how to increase or modify this limit. I just found out this question can be considered a duplicate of this other unanswered question: Upload large file to google drive with PHP Client Library

I'm trying to upload a big file (237MB) to my google drive account from my VPS using the PHP CLI script provided on google API's quickstart guide and I get the following error:

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted 
(tried to allocate 316952783 bytes) in 
/var/www/google-api-php-client/src/service/Google_MediaFileUpload.php on line 135

The script:

<?php
require_once 'google-api-php-client/src/Google_Client.php';
require_once 'google-api-php-client/src/contrib/Google_DriveService.php';

$client = new Google_Client();
// Get your credentials from the APIs Console
$client->setClientId('MY_CLIENT_ID');
$client->setClientSecret('MY_CLIENT_SECRET');
$client->setRedirectUri('urn:ietf:wg:oauth:2.0:oob');
$client->setScopes(array('https://www.googleapis.com/auth/drive'));

$service = new Google_DriveService($client);

$authUrl = $client->createAuthUrl();

//Request authorization
print "Please visit:\n$authUrl\n\n";
print "Please enter the auth code:\n";
$authCode = trim(fgets(STDIN));

// Exchange authorization code for access token
$accessToken = $client->authenticate($authCode);
$client->setAccessToken($accessToken);

//Insert a file
$file = new Google_DriveFile();
$file->setTitle('My zip file');
$file->setDescription('A test zip file');
$file->setMimeType('application/zip');

$data = file_get_contents('myfile.zip');

$createdFile = $service->files->insert($file, array(
      'data' => $data,
      'mimeType' => 'application/zip',
    ));

print_r($createdFile);
?>

top says I've got 787816k physical memory free (plus 2G cache), so I don't get where the problem is. Any clues?

I think that's wrong, there's no duplicate here. The memory allocated is 1G, as the error specifies... — NotGaeL, Mar 04 '13 at 23:03
It looks like PHP is passing a copy of the `$data` when you call `$service->files->insert()`. You may be able to force PHP to pass the `$data` as a reference so it doesn't make a copy. If the internals of the API don't force a copy to be made, this could significantly reduce your memory usage. — G-Nugget, Mar 04 '13 at 23:05
It worked! Thank you @G-Nugget! (Anyway if anybody knows a method which doesn't require to load the entire file at once please share ;)) — NotGaeL, Mar 04 '13 at 23:09
I yelled victory too soon: Same output, only this time delayed a few seconds (I thought it was transferring) — NotGaeL, Mar 04 '13 at 23:18
First reset your memory limit to the original value, then have a look at this: https://developers.google.com/drive/manage-uploads#resumable — Michel Feldheim, Mar 05 '13 at 00:20
@elcodedocle sorry about that. You are right, I misread the value. — Tchoupi, Mar 05 '13 at 00:28
found a duplicate, it's unanswered: http://stackoverflow.com/questions/13808528/upload-large-file-to-google-drive-with-php-client-library — NotGaeL, Mar 05 '13 at 09:17
found another duplicate, this time with a valid answer: http://stackoverflow.com/a/14693917/1060686 — NotGaeL, Mar 10 '13 at 11:37

score 3 · Accepted Answer · edited May 23 '17 at 11:57

3

[EDIT] This is a valid answer to the memory allocation problem, but does not solve file upload within reasonable script memory usage constrains, which was the actual question. For an answer to that question, check out https://stackoverflow.com/a/14693917/1060686

PHP has a mechanism that prevents PHP scripts from using more memory than allowed by a given threshold. This threshold can be configured in the php.ini (for all scripts) or per script. I would advice to set it per script to a higher value.

On top of your script add the following line:

ini_set('memory_limit', '500M');

Explanation of large memory usage: You read the file using the following line:

$data = file_get_contents('myfile.zip');

This requires PHP to read the full file contents into memory ( in var $data ). It would be better to read the file blockwise e.g. in 4096 byte blocks and send them to network immediately. Like I said, I currently don't know if its possible using the google API that you are using.

edited May 23 '17 at 11:57

Community

1
1

answered Mar 04 '13 at 22:57

hek2mgl

152,036
28
249
266

It looks like the current `memory_limit` is `1G`, more than the suggested `500M`. – G-Nugget Mar 04 '13 at 23:02
Already set to 1G, as the error specifies. Tried your suggestion anyway. Still not working :( – NotGaeL Mar 04 '13 at 23:02
Ok, understood. Hmmm its hard to say, without testing the code. (will do, but will take some time) Isn't it possible to pass just the filename to the google api code? – hek2mgl Mar 04 '13 at 23:07
Not that I know of, and I was looking for it for a while, so if anybody is familiar with any method which does not require loading the entire file at once I'll be truly grateful to receive that knowledge :) – NotGaeL Mar 04 '13 at 23:12
Can you point me to (fast) way to obtain that client id and secret? I have a) logged into google, b) actived drive api c) m i s s i n g – hek2mgl Mar 04 '13 at 23:17
Here https://developers.google.com/drive/quickstart-php#step_1_enable_the_drive_api explains it. Go to https://code.google.com/apis/console/ and then select `api access` on the left menu and click on the big blue button labeled `Create an OAuth 2.0 client ID`. You only have to fill any name for you app and in the next sections choose `Installed application` for the Application type and `Other` for the Installed application type. – NotGaeL Mar 04 '13 at 23:21
make sure you replaced `$client->setClientId('MY_CLIENT_ID');` for `$client->setClientId('mybigassidnumber.apps.googleusercontent.com');` and `$client->setClientSecret('MY_CLIENT_SECRET');` for `$client->setClientSecret('mybigasspassword');` that you just obtained, otherwise I don't know what could be causing the problem – NotGaeL Mar 04 '13 at 23:29
1

At the moment I just can say that the behaviour is reproducible. Your example follows the example on the google website. So it seems like a bug to. May I'll dig more into it but it takes some effort and will take time. Maybe tomorrow I'll know more.... – hek2mgl Mar 05 '13 at 00:17
1

found a duplicate, with an answer: http://stackoverflow.com/a/14693917/1060686 – NotGaeL Mar 10 '13 at 11:36

Memory allocation error on basic Google Drive PHP API upload script

1 Answers1