Unzip POST body with node + express

Question

I've a simple node app that should write metrics from clients. Clients send metrics in json format zipped with python's zlib module, I'm trying to add a middleware to unzip the request post before the express bodyParse takes place.

My middlewares are simply the ones provided by express by default:

app.configure(function(){
    app.set('port', process.env.PORT || 3000);
    app.set('views', __dirname + '/views');
    app.set('view engine', 'jade');
    app.use(express.favicon());
    app.use(express.logger('dev'));
    app.use(express.bodyParser());
    app.use(express.methodOverride());
    app.use(express.cookieParser('your secret here'));
    app.use(express.session());
    app.use(app.router);
    app.use(require('less-middleware')({ src: __dirname + '/public' }));
    app.use(express.static(path.join(__dirname, 'public')));
});

I've tried to add a simple middleware that gets the data and then unzips it:

app.use(function(req, res, next) {
    var data = '';
    req.addListener("data", function(chunk) {
        data += chunk;
    });

    req.addListener("end", function() {
        zlib.inflate(data, function(err, buffer) {
            if (!err) {
                req.body = buffer;
                next();
            } else {
                next(err);
            }
        });
    });
});

The problem is with zlib.inflate I get this error:

Error: incorrect header check

The data has been compressed with python's zlib module:

zlib.compress(jsonString)

but seems that neither unzip, inflate, gunzip works.

Are you sure you don't get an "err" from zlib.unzip and as a result the next() is not called? If that's the case, try switching the data an array. Check out this answer http://stackoverflow.com/a/12776856/30763 In addition make sure your middleware is used before bodyParser. — Aaron, Jan 07 '13 at 12:54
That answer isn't for unzipping a request done by node and not by the client? — alex88, Jan 07 '13 at 13:27
@alex88 in any case, you should call `next(err)` if there is an error. I also suggest that you forgo the `bodyParser` middleware and just do `req.body = JSON.parse(buffer)` directly. — Linus Thiel, Jan 07 '13 at 13:34
Ok so the problem is with the unzip function, thanks for pointing that out, I've corrected the question now. — alex88, Jan 07 '13 at 13:46

score 13 · Accepted Answer · answered Jan 07 '13 at 14:59

13

Found the solution on my own, the problem was with this piece of code:

req.addListener("data", function(chunk) {
    data += chunk;
});

seems that concatenating request data isn't correct, so I've switched my middleware to this:

app.use(function(req, res, next) {
    var data = [];
    req.addListener("data", function(chunk) {
        data.push(new Buffer(chunk));
    });
    req.addListener("end", function() {
        buffer = Buffer.concat(data);
        zlib.inflate(buffer, function(err, result) {
            if (!err) {
                req.body = result.toString();
                next();
            } else {
                next(err);
            }
        });
    });
});

concatenating buffers works perfectly and I'm now able to get request body decompressed.

answered Jan 07 '13 at 14:59

alex88

4,788
4
39
60

In case you're wondering *why*, it's because your first attempt was converting raw binary data into a string. Since UTF-8 isn't meant to hold arbitrary bytes (like those you'd find in a compressed data stream), your data was corrupted. By default, the [`data`](http://nodejs.org/api/http.html#http_event_data) event gives you a [`Buffer`](http://nodejs.org/api/buffer.html#buffer_buffer), which is Node's way of dealing with raw data. – josh3736 Jan 07 '13 at 15:14
@josh3736 and converting to string was losing some data? So the chunk is already a Buffer and I can easily push that instead of creating a new Buffer? – alex88 Jan 07 '13 at 15:46
[Yes. From the docs for the `data` event:](http://nodejs.org/api/http.html#http_event_data) "*The `chunk` is a string if an encoding has been set with `request.setEncoding()`, otherwise it's a `Buffer`.*" – josh3736 Jan 07 '13 at 15:50
hey, can you post the complete app initialization code?I'm facing a problem where the callback of the inflate (i'm using gunzip) my result is empty – Tal Barda Dec 30 '15 at 08:40

score 3 · Answer 2 · answered Apr 29 '15 at 07:51

I know this is a very late response, but with module body-parser, it will:

Returns middleware that only parses json. This parser accepts any Unicode encoding of the body and supports automatic inflation of gzip and deflate encodings.

var bodyParser = require('body-parser');
app.use( bodyParser.json() );       // to support JSON-encoded bodies

Unzip POST body with node + express

2 Answers2

Linked