How can I gunzip POST request data in express?

Question

I am trying to build a server that can accept gzipped POST data with express. I think I could just write my own middleware to pipe the request stream to a zlib.createGunzip() stream. The question is, how can I achieve that, afterwards, the express.bodyParser() middleware is still able to parse my gunzipped POST data?

I tried to replace the original request stream methods by the ones of the zlib stream, but that just made the bodyParser return a "Bad Request" Error:

var express = require('express');
var app = express();

function gUnzip(req, res, next) {
  var newReq;
  if (req.headers['content-encoding'] === 'gzip') {
    console.log("received gzipped body");
    newReq = req.pipe(zlib.createGunzip());
    Object.getOwnPropertyNames(newReq).forEach(function (p) {
      req[p] = newReq[p];
    });
  }
  next();
}

app.use(gUnzip);
app.use(express.bodyParser());

app.listen(8080);

Is there a way to make this work without rewriting the bodyParser() middleware within my own middleware?

EDIT: This is the same question: Unzip POST body with node + express. But in the answer he just does in his own middleware what the express.bodyParser() should do, which is what I want to avoid. I am looking for a way to simply unzip the request data from the stream and then pass it to the bodyParser(), which expects a stream itself, as can be seen at http://www.senchalabs.org/connect/json.html.

If you do `delete req.headers['content-encoding']`, you won't get a `Bad Request` anymore, but it still doesn't work :( — robertklep, Apr 03 '13 at 09:59
As far as I can see, you can't 'overwrite' the request object like that. I think you're going to need to implement `bodyParser` functionality yourself :( — robertklep, Apr 03 '13 at 10:16

score 4 · Answer 1 · answered Mar 05 '14 at 06:04

compressed request bodies are generally not used because you can't negotiate content encodings between the client and server easily (there's another stackoverflow question about that i believe). most servers don't support compressed request bodies, and the only time you really need it is for APIs where the client will send large bodies.

body-parser, specifically raw-body, does not support it because the use-case is so minimal, though i've though about adding it. for now, you'll have to create your body-parser. fortunately, that's easy since you can just fork body-parser and leverage raw-body. the main code you'll add around https://github.com/expressjs/body-parser/blob/master/index.js#L80:

    var zlib = require('zlib')

        var stream
        switch (req.headers['content-encoding'] || 'identity') {
            case 'gzip': 
                stream = req.pipe(zlib.createGunzip())
                break
            case 'deflate': 
                stream = req.pipe(zlib.createInflate())
                break
            case 'identity': 
                break
            default:
                var err = new Error('encoding not supported')
                err.status = 415
                next(err)
                return
        }

        getBody(stream || req, {
            limit: '1mb',
            // only check content-length if body is not encoded
            length: !stream && req.headers['content-length'],
            encoding: 'utf8'
        }, function (err, buf) {

        })

score -3 · Answer 2 · edited May 23 '17 at 12:08

-3

~~Have you tried using the built in compress middleware. It's documented in the expressjs reference documentation~~

app.use(express.compress());

Maybe you can find something useful here instead: Unzip POST body with node + express

edited May 23 '17 at 12:08

Community

1
1

answered Apr 03 '13 at 09:56

NilsH

13,705
4
41
59

yes, but express.compress() only supports gzipping the server _responses_. I want to send a gzipped POST _request_ from a client and unzip it on the server. – Steve Beer Apr 03 '13 at 09:57
Aha. I missed that part. Sorry about the confustion. – NilsH Apr 03 '13 at 10:01
Maybe the same question: http://stackoverflow.com/questions/14194864/unzip-post-body-with-node-express – NilsH Apr 03 '13 at 10:09
1

The solution there doesn't seem to work with `express.bodyParser()` properly, since that will also try to read the request stream, which the gunzipping middleware just did as well. The result is `bodyParser()` hanging (that could be circumvented by setting `req._body` is true when the stream has been read, but that would make `bodyParser` skip handling the body entirely). – robertklep Apr 03 '13 at 10:15

How can I gunzip POST request data in express?

2 Answers2