1

I'm trying to fetch a large JSON in Google Script but end up with a truncated response since it hit the 50Mb maximum fetch response size. So I'm now trying to get it compressed since in my browser the request takes only 12Mb but can't find a way to make it work, any idea?

var response = UrlFetchApp.fetch(URL, {
    method: "GET",
    headers: {
      'accept-encoding': 'gzip',
      "User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
    }
});

var body = response.getContentText();           // Return the full JSON, should have been compressed
console.log(body.length);                       // Return 50Mb
console.log(body.substring(body.length-50));    // Can confirm it's truncated and not encoded

var res = Utilities.ungzip(response.getBlob()); // Error Invalid argument

I'm not getting the compressed response and I'm getting Error Invalid argument on the last line.

I tried some other ways but still can't get a compressed response.

TnK
  • 21
  • 6
  • See [mcve]. *Describe the problem. "It doesn't work" isn't descriptive enough to help people understand your problem. Instead, tell other readers what the expected behavior should be. Tell other readers what the exact wording of the error message is, and which line of code is producing it. Use a brief but descriptive summary of your problem as the title of your question.* – TheMaster Aug 17 '20 at 04:51
  • All of this is provided in my question, I don't get what's wrong with it – TnK Aug 17 '20 at 07:48
  • Is there a error? Which part of the code doesn't work? What do you mean by 50kb(Did you mean Mb)? – TheMaster Aug 17 '20 at 08:46
  • Shouldn't you contact your server? Could you show browser's request headers? – TheMaster Aug 17 '20 at 08:54
  • Oh okay sorry about the 50kb, yes it was Mb (my bad) and I was refering to [fetch response quota](https://developers.google.com/apps-script/guides/services/quotas). The rest of the question is pretty clear imo, I'm trying to get my response as a gzip to not hit that quota but I never get a compressed response and end up with a truncated JSON. – TnK Aug 17 '20 at 09:14
  • Does the server support gzip? – TheMaster Aug 17 '20 at 10:35
  • Well, I guess so since the request initiated within chrome is 12MB and has header `Accept-Encoding: gzip, deflate, br` That being said, the response header has `content-encoding: br` so maybe I'm wrong here. – TnK Aug 17 '20 at 11:59
  • 1
    So why are you only sending `gzip`. Emulate everything the browser sends with `deflate, br` etc. What does the `Accept` header look like? – TheMaster Aug 17 '20 at 12:04
  • Using the headers `'accept-encoding': 'gzip, deflate, br','Accept': '*/*'` like the request initiated from chrome gives me the same response. Response headers contain `Content-Encoding=br,Content-Type=application/json` – TnK Aug 17 '20 at 12:20
  • same response as in compressed response? – TheMaster Aug 17 '20 at 12:55
  • Just like the code in my question : `console.log(response.getContentText())` display the json uncompressed, `console.log(response.getContentText().length)` is 50MB, `console.log(str.substring(str.length-50))` show it being truncated. It's just not compressed, or I'm really missing something. – TnK Aug 17 '20 at 13:48
  • You can check for content-encoding: `console.log(response.getAllHeaders()["content-encoding"])` to see if gzip negotiation is complete and your request is accepted by the server – TheMaster Aug 17 '20 at 14:44
  • Alright so I don't know if the code is case sensitive here, but in the response _in Google Scripts_, my header is named **Content-Encoding** and it's value is **br**. _In Firefox_ the header value is also **br** but the header is named **content-encoding** without capital letters. – TnK Aug 17 '20 at 16:24
  • Doesn't matter. If the header is br, then the server seems to be responding with compressed data. Don't know if it is automatically uncompressed by urlfetch. Try curl to check content length in both cases. – TheMaster Aug 17 '20 at 19:05
  • I don't know, I'm running `curl -H "Accept-Encoding: gzip,deflate,br" -I URL` and it doesn't return me any `Content-Length` header... Here is what I get, among others but unrelated headers `Content-Type: application/json Access-Control-Expose-Headers: Content-Length,Content-Range Content-Encoding: br`. I'm lost tbh. – TnK Aug 18 '20 at 19:16
  • ok what about the response itself? Does it look like it is compressed or is it's size reduced? – TheMaster Aug 19 '20 at 04:45
  • No it was all clear, nothing compressed. At the end I gave up and created a new JSON with the truncated response, I'm losing like 5% of what the response should be but it's not critical and I'm just wasting too much time on this :/ – TnK Aug 19 '20 at 08:08
  • 1
    If you feel like wasting more time and your server supports partial downloads, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range and https://stackoverflow.com/questions/60181888 – TheMaster Aug 19 '20 at 11:11

0 Answers0