Easy HTTP requests with gzip/deflate compression

Question

I'm trying to figure out how the best way to easily send HTTP/HTTPS requests and to handle gzip/deflate compressed responses along with cookies.

The best I found was https://github.com/mikeal/request which handles everything except compression. Is there a module or method that will do everything I ask?

If not, can I combine request and zlib in some manner? I tried to combine zlib and http.ServerRequest, and it failed miserably.

Answer 1

For anyone coming across this in recent times, the request library supports gzip decompression out of the box now. Use as follows:

request(
    { method: 'GET'
    , uri: 'http://www.google.com'
    , gzip: true
    }
  , function (error, response, body) {
      // body is the decompressed response body
      console.log('server encoded the data as: ' + (response.headers['content-encoding'] || 'identity'))
      console.log('the decoded data is: ' + body)
    }
  )

From the github readme https://github.com/request/request

gzip - If true, add an Accept-Encoding header to request compressed content encodings from the server (if not already present) and decode supported content encodings in the response. Note: Automatic decoding of the response content is performed on the body data returned through request (both through the request stream and passed to the callback function) but is not performed on the response stream (available from the response event) which is the unmodified http.IncomingMessage object which may contain compressed data. See example below.

Answer 2

Note: as of 2019, request has gzip decompression built in. You can still decompress requests manually using the below method.

You can simply combine request and zlib with streams.

Here is an example assuming you have a server listening on port 8000 :

var request = require('request'), zlib = require('zlib');

var headers = {
    'Accept-Encoding': 'gzip'
};

request({url:'http://localhost:8000/', 'headers': headers})
    .pipe(zlib.createGunzip()) // unzip
    .pipe(process.stdout); // do whatever you want with the stream

Answer 3

Here's a working example that gunzips the response

function gunzipJSON(response){

    var gunzip = zlib.createGunzip();
    var json = "";

    gunzip.on('data', function(data){
        json += data.toString();
    });

    gunzip.on('end', function(){
        parseJSON(json);
    });

    response.pipe(gunzip);
}

Full code: https://gist.github.com/0xPr0xy/5002984

Answer 4

Check out the examples at http://nodejs.org/docs/v0.6.0/api/zlib.html#examples

zlib is now built into node.

Answer 5

Looking inside the source code - you must set the gzip param on the request lib itself for gzip to work. Not sure if this was intentional or not, but this is the current implementation. No extra headers are needed.

var request = require('request');
request.gzip = true;
request({url: 'https://...'},  // use encoding:null for buffer instead of UTF8
    function(error, response, body) { ... }
);

Answer 6

All the answers here did not work and I was getting raw bytes back instead and the gzip flag was not working either. As it turns out you need to set the encoding to null to prevent requests from transforming the response to utf-8 encoding and instead keeps the binary response.

const request = require("request-promise-native");
const zlib = require("zlib");

const url = getURL("index.txt");
const dataByteBuffer = await request(url, { encoding: null });
const dataString = zlib.gunzipSync(response);