nginx logging $request_body as hexadecimal

Question

I am trying to log the request body of requests to my api and nginx is turning all quotes (and some other characters like spaces and tabs) into hexadecimal characters.

Here is my log format

log_format postdata '{"ts": "$time_iso8601", "status": $status, "req": "$uri", "meth": "$request_method", "body": "$request_body"}';

Here is what gets logged

{"ts": "2015-05-20T15:31:11-07:00", "status": 400, "req": "/v2/track", "meth": "POST", "body": {\x22id\x22:\x22user id\x22}}

How can I prevent this so that the resulting log line is

{"ts": "2015-05-20T15:31:11-07:00", "status": 400, "req": "/v2/track", "meth": "POST", "body": {"id":"user id"}}

You can't. It's harcoded in nginx's log module. But if you put quotes around (like you show in log format, but not in actual result) it would be valid js string, which you could use in `JSON.parse` — Alexey Ten, May 21 '15 at 07:40
Well, not actually valid if you use non-ascii, but nginx's log module was written long before JSON became popular or even existed — Alexey Ten, May 21 '15 at 07:41
Related question which sheds a little insight into this behaviour can be found [here](http://serverfault.com/q/247271) — TomDotTom, Oct 04 '16 at 14:35
Hi @tbeauvais, now there's a real solution to this question - please accept it. See the answer from Grigori Kochanov. — noamtm, Aug 12 '20 at 12:45

score 27 · Accepted Answer · answered Aug 29 '19 at 17:34

27

Sinse 1.13 there is an "escape=none" parameter that turns off data escaping.

http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

log_format  api_request_log escape=none '[$time_local] $request \n$request_body';

answered Aug 29 '19 at 17:34

Grigori Kochanov

794
7
7

1

It's 2020, and this is definitely the correct answer. – noamtm Aug 12 '20 at 13:15
There's also an `escape=json` [as of 1.11.8](https://github.com/nginx/nginx/commit/c40d8dd). – sengi Jul 07 '23 at 09:23
Is there a reason for using `escape=none` when outputting JSON? Wouldn't `escape=none` mean that JSON metacharacters in request fields (i.e. from network input) could mess with the JSON syntax of the log output? – sengi Jul 07 '23 at 09:30

score 9 · Answer 2 · edited May 23 '17 at 12:32

You can't stop from escaping it and will have to post process it.

Python2 example:

line = '{\x22id\x22:\x22user id\x22}'
line.decode('unicode_escape')
>> u'{"id":"user id"}'

Python3 example:

line = '{\x22id\x22:\x22user id\x22}'
bytes(line, 'utf-8').decode('unicode_escape')
>> '{"id":"user id"}'

Ruby example (from https://stackoverflow.com/a/18752208/2398354):

require 'yaml'
line = '{\x22id\x22:\x22user id\x22}'
YAML.load(%Q(---\n"#{line}"\n))
=> "{\"id\":\"user id\"}"

Note: This last example is useful if post processing a file with logstash

score 9 · Answer 3 · answered Jul 21 '17 at 04:58

9

Hope this will be helpful for someone. In order to log entire json request unescaped. Do add in http block this configuration

http {
 log_format postdata escape=json $request_body;
 access_log /var/log/nginx/access.log postdata;
 .....
}

answered Jul 21 '17 at 04:58

Neftanic

930
9
17

this added an additional string literal to my logs `escape=json{\x22d\x22:\x22d\x22}` – Jayson Cheng Dec 04 '17 at 11:22
i got it. only nginx <= 1.11.8 supports this – Jayson Cheng Dec 05 '17 at 04:30
@JaysonCheng **>=** 1.11.8, right? – sengi Jul 07 '23 at 09:27

score 0 · Answer 4 · answered Mar 15 '17 at 18:37

Like others said, there is no way to fix this within nginx configuration. But it is not difficult to post-process it. If the request body is JSON-formatted, you'll probably run into a lot of \x0A (newline) and \x22 (").

Just clear those out with sed before looking into the logfile.

Here is the command for you: LANG='' sed -E 's/(\\x0A|\\x22)//g' access.log

nginx logging $request_body as hexadecimal

4 Answers4