nginx logs to redis to logstash

Question

I want send nginx log files(on one server) to redis(on 2nd server) in order to process it later using logstash(on 3rd server), but I am kinda lost

I am using redis as buffer not to lose any data in case logstash server gone down for maintenance or any other reason

Can you add some of you code / configuration? – Athafoud Oct 11 '16 at 12:52 — Athafoud, Oct 11 '16 at 12:52

score 3 · Answer 1 · answered Sep 18 '17 at 04:41

So this is how I solved it, I made nginx to log to syslog server and from syslog-ng to redis here is my configuration in nginx http directive

log_format xxx_log_format '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $bytes_sent '
                       '"$http_referer" "$http_user_agent" "$gzip_ratio"';

in nginx server directive

access_log syslog:server=127.0.0.1:601 xxx_log_format;

in syslog-ng config

source s_syslog {
    udp(
        port(601)
    );
};

destination d_redis {
    redis(
        host("REDIS-IP")
        port(6379)
        command("LPUSH", "access_logs", "${MESSAGE}")
    );
};

log {
    source(s_syslog);
    destination(d_redis);
};

score 0 · Answer 2 · answered Oct 11 '16 at 16:29

0

Take a look to the docs, there is an example for managing spikes with message queueing:

https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#deploying-message-queueing

Logstash input and output plugins should help.

answered Oct 11 '16 at 16:29

Janusz

1,343
11
16

score 0 · Answer 3 · answered Oct 11 '16 at 17:57

I use a small logstash instance to receive events and write them to redis, and then a second one to read from redis, do all the magic processing, and write to elasticsearch.

I only send events that might normally be lost to the first instance - syslog, snmptrap, etc (where the sender is just blindly sending events). I send filebeat traffic directly to the second logstash instance, as filebeat will stop sending if it's down. (As long as logstash isn't down during a client-side log file rotation, you'll be OK).

score 0 · Answer 4 · answered Sep 16 '17 at 11:59

if you want to log real time to redis you can use redis module for nginx - https://www.nginx.com/resources/wiki/modules/redis2/

if you want to just delay the redis call to background to preserve fast response time - use undocumeted directive of nginx called post_action - google it.

This is in case that you are not trusting logstash and want to solve it using redis.

I can suggest alternatives that you can have a look on:

fluentd redis output plugin and any other plugin hou want like elastic (similar to logstash)
you can make an nginx log rotatation very hour or so and compare it to output copied from output plugin of fluentd or logstash to wherever (e.g. s3) and make a diff in some way (based on hours of in the log file, using sort and comm linux commannds or emr or athena) then you can have log the diff sent back to whatever target needed like logtash, fluentd or elk.

nginx logs to redis to logstash

4 Answers4