2

I currently see

172.19.0.1 - - [09/Feb/2018:07:00:32 +0000] "GET /ping HTTP/1.1" 200 1 "-" "curl/7.47.0"

within my logs, but I use structured logging like this.

I even tried

ch = logging.StreamHandler()
ch.setFormatter(pythonjsonlogger.jsonlogger.JsonFormatter())
logging.getLogger("urllib3").addHandler(ch)

but I still see those messages. I have nginx/gunicorn like this (source):

nginx = subprocess.Popen(['nginx', '-c', '/opt/program/nginx.conf'])
gunicorn = subprocess.Popen(['gunicorn',
                             '--timeout', str(model_server_timeout),
                             '-k', 'gevent',
                             '-b', 'unix:/tmp/gunicorn.sock',
                             '-w', str(model_server_workers),
                             'server.wsgi:app'])

I guess this is where the log messages come from. But I have no idea how to get structured logging there.

Martin Thoma
  • 124,992
  • 159
  • 614
  • 958

2 Answers2

2

Found it. I had to edit the nginx.conf:

Within http:

log_format structured '{"remote_addr": "$remote_addr", "remote_user": "$remote_user", "request": "$request", "status": $status, "body_bytes_sent": $body_bytes_sent, "http_referer": "$http_referer", "http_user_agent": "$http_user_agent"}';

and within http > server:

access_log /dev/stdout structured;

See http://nginx.org/en/docs/http/ngx_http_log_module.html

Now the messages look like this:

{
    "remote_addr": "10.32.0.2",
    "remote_user": "-",
    "request": "GET /ping HTTP/1.1",
    "status": 200,
    "body_bytes_sent": 1,
    "http_referer": "-",
    "http_user_agent": "AHC/2.0"
}

Still TODO

It would be nice if the request was 

{"method": "GET",
 "endpoint": "/ping",
 "protocol": "HTTP/1.1"}
Martin Thoma
  • 124,992
  • 159
  • 614
  • 958
  • 1
    You can answer the `TODO` part with the following variables: ```{ "method": "$request_method", "endpoint": "$request_uri", "protocol": "$server_protocol" }``` – Samy Mar 14 '23 at 13:10
1

The logging format set in Python process has nothing to do with the one in Nginx and Gunicorn.

If you want to set Nginx logging format, see

https://www.nginx.com/resources/admin-guide/logging-and-monitoring/#access_log

For gunicorn, see http://docs.gunicorn.org/en/stable/settings.html#access-log-format

Jacky1205
  • 3,273
  • 3
  • 22
  • 44
  • From the log message above: What is the part created by gunicorn? How do I make it a structured log? – Martin Thoma Feb 09 '18 at 07:36
  • I guess the above log is captured by gunicorn, looks very like the default format. set `--access-logformat` to `'{"remote_addr": %(h)s...}'` as you like – Jacky1205 Feb 09 '18 at 08:27
  • + link http://docs.gunicorn.org/en/stable/settings.html#access-log-format – Jacky1205 Feb 09 '18 at 08:28
  • look at my answer. As I almost completely solved it with nginx config, it for sure is mostly nginx. only the "request" part might be gunicorn – Martin Thoma Feb 09 '18 at 08:48