Nginx gunzip POST requests to backend

Question

I have plenty of HTTP POST requests being sent to a nginx server that then get load balanced to a set of reversed proxied node.js/express.js backend servers. To spare some network consumption the payload is being sent using GZIP and header Content-Encoding: gzip.

I'm trying to achieve something like this:

[Client] [Nginx Reverse Proxy] [BackEnd] | [gziped payload] | [raw payload] |
|--------------------> | ----------------------------->| | | |
| [Raw Response] | [Raw response] | | <------------------ | <-----------------------------| | | |

For efficiency reasons i want to run Gunzip at Nginx yet i haven't been able to do so. Here's the http nginx.conf file:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;
        gunzip on;
        location / {
           proxy_pass http://localhost:8080;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_cache_bypass $http_upgrade;
          }
    }
}

Here's an example of a request:

echo "{"id": 0,"mypayload":"This is an example"}" | gzip -c - | curl -v -i -X POST -H 'Content-Encoding: gzip' --data-binary '@-' http://localhost/test

I wanted nginx to unzip the content of the payload and deliver the raw content to the backend server, yet the content is still being delivered compressed.

I've seen plenty of folks doing the other way around (Nginx gziping responses and even gunzipping responses to clients that do not have the Accept-Encoding: gzip header) yet couldn't find it successfully gunziping payloads to the backend.

Any clues?

Answer 1

I know this is an older question, but I was looking to do the same for an IOS client that sends compress requests and after checking out your question I came upon a lua solution described here and thought to post it here for future reference.

The lua module looks like:

-- Debian packages nginx-extras, lua-zlib required

ngx.ctx.max_chunk_size = tonumber(ngx.var.max_chunk_size)
ngx.ctx.max_body_size = tonumber(ngx.var.max_body_size)

function create_error_response (code, description)
    local message = string.format('{"status":400,"statusReason":"Bad Request","code":%d,"exception":"","description":"%s","message":"HTTP 400 Bad Request"}', code, description)
    ngx.status = ngx.HTTP_BAD_REQUEST
    ngx.header.content_type = "application/json"
    ngx.say(message)
    ngx.exit(ngx.HTTP_OK)
end


function inflate_chunk (stream, chunk)
    return stream(chunk)
end


function inflate_body (data)
    local stream = require("zlib").inflate()
    local buffer = ""
    local chunk = ""

    for index = 0, data:len(), ngx.ctx.max_chunk_size do
        chunk = string.sub(data, index, index + ngx.ctx.max_chunk_size - 1)
        local status, output, eof, bytes_in, bytes_out = pcall(stream, chunk)

        if not status then
            -- corrupted chunk
            ngx.log(ngx.ERR, output)
            create_error_response(4001, "Corrupted GZIP body")
        end

        if bytes_in == 0 and bytes_out == 0 then
            -- body is not gzip compressed
            create_error_response(4002, "Invalid GZIP body")
        end

        buffer = buffer .. output

        if bytes_out > ngx.ctx.max_body_size then
            -- uncompressed body too large
            create_error_response(4003, "Uncompressed body too large")
        end
    end

    return buffer
end


local content_encoding = ngx.req.get_headers()["Content-Encoding"]
if content_encoding == "gzip" then
    ngx.req.read_body()
    local data = ngx.req.get_body_data()

    if data ~= '' then
        local new_data = inflate_body(data)

        ngx.req.clear_header("Content-Encoding")
        ngx.req.clear_header("Content-Length")
        ngx.req.set_body_data(new_data)
    end
end

and then you can use it like:

location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_pass_request_headers on;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;

        client_max_body_size 512k;    # Max request body size of 512 KB
        client_body_buffer_size 512k;

        set $max_chunk_size = 10240;  # Chunks of 10 KB
        set $max_body_size = 524288;  # Max inflated body size of 512 KB

        rewrite_by_lua_file inflate_body.lua;
    }