Node.js encoding issue

Question

I am trying to get data from an api (api.keepa.com, just to test) in my firebase cloud function. Therefore, I have this code:

var url = "https://api.keepa.com"

var request = require('request');

console.log(url)

request(url, function (error1, response1, body1) {
    if (!error1 && response1.statusCode == 200) {
        console.log(body1)
        }
    else{
        console.log("Error:",error1)
    }
});

When I check it in the browser, I get a normal result ("Interested in our data? Check out our API: https://keepa.com/#!api").

When I run it in firebase cloud function or in repl.it, I just get "��+I-J-.IMQ��S�/-RHI,I�Wp�HM��K�b��V%%�V��٩��z��ʊ��\&��H".

Is there anything I am doing wrong? Looks like a encoding issue for me.

Update: I added a screenshot of the header data.

For the real api access it is a json - but when I do this request I have the same issues, so I thought to testet with the basic request — Alexander Ko, Apr 10 '18 at 20:23
I mean to specifically check the headers of the response to see what you're getting. It might be compressed and you're just failing to uncompress it. Your browser will silently decompress this for you meaning you won't notice unless you look carefully. — tadman, Apr 10 '18 at 20:26
Possible duplicate of [node.js - easy http requests with gzip/deflate compression](https://stackoverflow.com/questions/8880741/node-js-easy-http-requests-with-gzip-deflate-compression) — zero298, Apr 10 '18 at 20:52

zero298 · Accepted Answer · 2018-04-10T20:40:18.687

Tell request that you want it to decompress the response with gzip:

const args = {
  uri: "https://api.keepa.com",
  gzip: true
};

request(args, (err, res, body) => err ? console.error(err) : console.log(body));

Per the docs:

gzip - if true, add an Accept-Encoding header to request compressed content encodings from the server (if not already present) and decode supported content encodings in the response. Note: Automatic decoding of the response content is performed on the body data returned through request (both through the request stream and passed to the callback function) but is not performed on the response stream (available from the response event) which is the unmodified http.IncomingMessage object which may contain compressed data. See example below.

score 0 · Answer 2 · answered Apr 10 '18 at 20:31

Look at the Content-Encoding header. It's gzipped, and in Node you have to run that through the Zlib function to uncompress it first before it'll be readable JSON data.

You can do this with streams more easily, where you pipe to the decompress stream, and then pipe that to your JSON decoder.

Alternatively you can make your request but set your Accept-Encoding header to refuse compression, though that's going to lead to slower downloads and more bandwidth used.

Node.js encoding issue

2 Answers2