Google apps script get range of bytes from binary file

Question

I'm aware that in the Google Drive API you can request to get a portion of a file within a certain range: Range: bytes=500-999, but I was looking in the Google Apps Script reference and I couldn't seem to find the equivalent function in there.

Does anyone have any ideas?

Tanaike · Accepted Answer · 2020-08-19T12:11:17.547

5

You want to retrieve the partial body of the file.
You want to achieve this using Google Apps Script.

If my understanding is correct, how about this answer? Please think of this as just one of several possible answers.

In order to achieve the partial download, please use Range: bytes=500-999 at the request headers.

Sample script:

As a simple sample script, how about the following script? In this script, the partial download from 100 bytes to 200 bytes is run for a file of file ID using Drive API.

var fileId = "###";  // Please set the file ID.

var start = 100;
var end = 199;
var url = "https://www.googleapis.com/drive/v3/files/" + fileId + "?alt=media";
var params = {
  method: "get",
  headers: {
    Authorization: "Bearer " + ScriptApp.getOAuthToken(),
    Range: "bytes=" + start + "-" + end,
  },
};
var bytes = UrlFetchApp.fetch(url, params).getContent();
Logger.log(bytes.length)

When you run the script, 100 can be seen at the log.
As one important point, the range of the 1st byte is 0.
As another important point, at Google Apps Script, the maximum size of the blob is 50 MB (52,428,800 bytes). So also please be careful this.

Note:

As one more sample script, how about this?

References:

If I misunderstood your question and this was not the direction you want, I apologize.

Updated at August 19:

When I tested the maximum size of the byte array, I could notice that the specification had been changed. So I add it as the updated specification.

Before: The maximum blob size is 52,428,800 bytes. Accurately, when the blob more than 50 MB is converted to the byte array, an error occurs. By this, when the byte array of 50 MB is added to the byte array of 50 MB, an error related to the maximum size occurs. This was my experimental result I measured before.
Now: Now, when I tested this situation again, although the file of 52,428,801 bytes cannot be retrieved as the byte array (this is the same specification. The maximum blob size is 52,428,800 bytes.), I noticed that the byte array of 52,428,800 bytes got to be able to be added to the byte array of 52,428,800 bytes. By this, I could confirm that the byte array of 104,857,600 could be created.

edited Aug 19 '20 at 12:11

answered Feb 12 '20 at 06:05

Tanaike

181,128
11
97
165

Thanks, ingenius, exactly what I was looking for! ( realized afterwards can just make the http request from within the google apps script). Just a quick note about the script you referenced: I noticed it involved storing a large file in a google spreadsheet. I may be interested in doing this to make a database, I would need to be able to #1 write chunks at a particular index and #2 read chunks at a particular index, do you know if this can be done with google spreadsheets? Meaning can I write, say, 4kb of data, at chunk position #14 (for example), and read a chunk at a chunk index? – B''H Bi'ezras -- Boruch Hashem Feb 12 '20 at 06:14
@bluejayke Thank you for replying. About the additional question, when the file is text data, your goal can be achieved by this method. In this case, I think that the partial data might be required to be parsed. On the other hand, when the file is the binary data, I think that when the partial download is run, the retrieved data might not be able to be decoded to the text data for putting to Spreadsheet. Please be careful this. – Tanaike Feb 12 '20 at 06:19
also just tested it, getting error: Request failed for https://www.googleapis.com returned code 403. Truncated server response: { "error": { "errors": [ { "domain": "global", "reason": "insufficientPermissions", "message": "Insufficient Permission: Request ... (use muteHttpExceptions option to examine full response) (line 3, file "Code") for file with ID: 1GZ21iYjMmfyOuiu6gXgasTH9q2_xxDPL – B''H Bi'ezras -- Boruch Hashem Feb 12 '20 at 06:25
@bluejayke Thank you for replying. I apologize for the inconvenience. From the error message of `insufficientPermissions`, it seems that the file cannot be retrieved. But unfortunately, I'm not sure about your actual situation. And I cannot understand about `line 3`. This is due to my poor skill. I deeply apologize for this. In order to correctly understand about your issue, can you provide the script and flow for replicating your issue? By this, I would like to try to replicate it. – Tanaike Feb 12 '20 at 06:39
Here's the scirpt: function myFunction() { var id = "1GZ21iYjMmfyOuiu6gXgasTH9q2_xxDPL"; var start = 4, end = 10, url = "https://www.googleapis.com/drive/v3/files/" + id + "?alt=media", tok = ScriptApp.getOAuthToken(); Logger.log(tok); var rez = UrlFetchApp.fetch(url, { method: "get", headers: { Authorization: " Bearer " + tok, Range: "bytes="+start+"-" +end } }); Logger.log(rez) } – B''H Bi'ezras -- Boruch Hashem Feb 12 '20 at 06:42
Do you have to be running the script (and getting the oauthtoken) from the same account that the file is in? – B''H Bi'ezras -- Boruch Hashem Feb 12 '20 at 06:46
@bluejayke Yes. Please use the file in your file in your Google drive. If you want to use the file in other account, please share it. About your script in the comment, please check it again. Unfortunately, it doesn't reflect my sample script. By the way, in my environment, I could confirm that when my file in my Google Drive and the access token using `ScriptApp.getOAuthToken()` are used, the script worked. – Tanaike Feb 12 '20 at 06:54
Yo I think I fixed it (just waiting for logs to show though), I just had to set the scopes manually as shown here: https://developers.google.com/drive/api/v3/reference/files/get?apix_params=%7B%22fileId%22%3A%221GZ21iYjMmfyOuiu6gXgasTH9q2_xxDPL%22%2C%22alt%22%3A%22json%22%7D and here https://developers.google.com/apps-script/concepts/scopes#setting_explicit_scopes but the code itself works! – B''H Bi'ezras -- Boruch Hashem Feb 12 '20 at 07:00
@bluejayke Thank you for replying. I'm glad your issue was resolved. I could study from your question. Thank you, too. – Tanaike Feb 12 '20 at 07:04
@Tanaike also I had a related question, how to use the google docs api within google apps script frmo the urlfetchapp to modify the docs etc., without using documentapp https://stackoverflow.com/questions/60387167/using-google-docs-api-from-within-google-apps-script (I want to do this so that I can use the ScriptApp.getOauthToken(), send it to my own server, and edit google docs directly, using the google apps script backend, without making my own API key which is subject to quotas etc.. is this possible? – B''H Bi'ezras -- Boruch Hashem Feb 25 '20 at 03:47
@bluejayke About your new question of https://stackoverflow.com/q/60387167 , I noticed that an answer has already been posted. I think that it will resolve your issue. If in your situation, it is difficult to link of Google Apps Script to Google Cloud Project, how about using Docs API of Advanced Google services as a workaround? In this case, the GAS project is not required to the cloud project, and the scopes can be set using the manifest. But I'm not sure about your detail situation. So if this was not the direction you want, I apologize. – Tanaike Feb 25 '20 at 04:17
@Tanaike LOL that actually works!!! wow amazing lol, you should post it as an answer (the other answer I'm not looking 4) – B''H Bi'ezras -- Boruch Hashem Feb 26 '20 at 01:48
@bluejayke Thank you for replying. I'm glad your issue was resolved. Your replying is that the method for using Docs API of Advanced Google services for [your this question](https://stackoverflow.com/q/60387167)? If it's so, I can propose the modified script. If I misunderstood your replying, I apologize. – Tanaike Feb 26 '20 at 02:18
ya thats the one – B''H Bi'ezras -- Boruch Hashem Feb 26 '20 at 02:41
1

*the maximum size of the blob is 50 MB (52,428,800 bytes).* The maximum fetch size is 50 OR the maximum blob size is 50? Can chunks of 50MB be downloaded and joint to create a big blob(say, 100MB)? – TheMaster Aug 19 '20 at 11:16
1

@TheMaster In this case, the maximum blob size is 52,428,800 bytes. Accurately, when the blob more than 50 MB is converted to the byte array, an error occurs. By this, when the byte array of 50 MB is added to the byte array of 50 MB, an error related to the maximum size occurs. This was my experimental result I measured before. – Tanaike Aug 19 '20 at 11:54
1

@TheMaster But now, when I tested this situation, although the file of 52,428,801 bytes cannot be retrieved as the byte array, I noticed that the byte array of 52,428,800 bytes got to be able to be added to the byte array of 52,428,800 bytes. By this, I could confirm that the byte array of 104,857,600 could be created. It was found that the specification of Google Apps Script had been changed. – Tanaike Aug 19 '20 at 11:54
@TheMaster Thank you, too. By your comment, I could have the chance to test it again. By this, we could notice that the specification had been changed. By the way, when you use UrlFetchApp, please be careful that the header data is also included in the maximum size. – Tanaike Aug 19 '20 at 12:02
1

For reference, I came from this: https://stackoverflow.com/questions/63443643 – TheMaster Aug 19 '20 at 12:06

Google apps script get range of bytes from binary file

1 Answers1

Sample script:

Note:

References:

Updated at August 19:

Linked