101

Let's say you have 2 very different types of logs such as technical and business logs and you want:

  • raw technical logs be routed towards a graylog2 server using a gelf output,
  • json business logs be stored into an elasticsearch cluster using the dedicated elasticsearch_http output.

I know that with Syslog-NG for instance, the configuration file allow to define several distinct inputs which can then be processed separately before being dispatched; what Logstash seems unable to do. Even if one instance can be initiated with two specific configuration files, all logs take the same channel and are being applied the same processings ...

Should I run as many instances as I have different types of logs?

David
  • 2,603
  • 4
  • 18
  • 28

3 Answers3

198

Should I run as many instances as I have different types of logs?

No! You can only run one instance to handle different types of logs.

In the logstash configuration file, you can specific each input with different type. Then in the filter you can use if to distinct different processing, and also at the output you can use "if" output to different destination.

input {
    file {
            type => "technical"
            path => "/home/technical/log"
    }
    file {
            type => "business"
            path => "/home/business/log"
    }
} 
filter {
    if [type] == "technical" {
            # processing .......
    }
    if [type] == "business" {
            # processing .......
    }
}
output {
    if [type] == "technical" {
            # output to gelf
    }
    if [type] == "business" {
            # output to elasticsearch
    }
}

Hope this can help you :)

Adam Michalik
  • 9,678
  • 13
  • 71
  • 102
Ban-Chuan Lim
  • 7,840
  • 4
  • 35
  • 52
  • 1
    You can also use the `type` attribute (with the same `type => "value"` syntax) in the filter and output definitions which should cut down on the extra formatting within the config file a bit. Example: https://gist.github.com/fairchild/3030472 Per documentation: Add a 'type' field to all events handled by this input. Types are used mainly for filter activation. The type is stored as part of the event itself, so you can also use the type to search for it in the web interface. – Tony Cesaro Mar 28 '14 at 22:09
  • 5
    Well, it looks like what Ben provided is actually the new way to do this. When I used `type => "value"` in an output, I got the following message displayed: "You are using a deprecated config setting "type" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. You can achieve this same behavior with the new conditionals, like: `if [type] == "sometype" { stdout { ... } }`." I retract my previous comment. :) – Tony Cesaro Mar 29 '14 at 02:56
  • 1
    Note that the `type` attribute won't apply if there's already a type field from the input. This is a special attribute that doesn't override and it's documented. I opened a ticket in Elastic and they recommended me to use `tags` or `add_field` instead of `type` – BornToCode Jun 18 '17 at 15:05
  • @BornToCode, I don't quite understand. Are you saying there is something wrong with Ben's code? Or that it wouldn't work if the log paths were the same file? What's the scenario where it doesn't work? – Ben Wheeler Sep 19 '17 at 17:07
  • @BenWheeler - I'm saying that in a scenario where there's already a "type" field (e.g. log came from filebeat service and filebeat was defined to add a custom field "type" with some value) then Ben's code won't work, because he checks `if [type] == "technical"` and that would never be true because type's value will always be the value that existed there before logstash came to play. – BornToCode Sep 24 '17 at 07:52
  • 2
    @BenLim The OP did not accept your answer but I found it most helpful and up-voted you. – bigbadmouse Jan 03 '19 at 16:15
  • Thank you. Although everyone says it's helpful but the OP do not want to accept my answer. lolx – Ban-Chuan Lim Jan 04 '19 at 01:01
18

I used tags for multiple file input:

input {
    file {
        type => "java"
        path => "/usr/aaa/logs/stdout.log"
        codec => multiline {
            ...
        },
        tags => ["aaa"]
    }

    file {
        type => "java"
        path => "/usr/bbb/logs/stdout.log"
        codec => multiline {
                ...
        }
        tags => ["bbb"]
    }
}
output {
    stdout {
        codec => rubydebug
    }
    if "aaa" in [tags] {
        elasticsearch {
            hosts => ["192.168.100.211:9200"]
            index => "aaa"
            document_type => "aaa-%{+YYYY.MM.dd}"
        }
    }

    if "bbb" in [tags] {
        elasticsearch {
            hosts => ["192.168.100.211:9200"]
            index => "bbb"
            document_type => "bbb-%{+YYYY.MM.dd}"
        }
    }
}
Robin Wang
  • 779
  • 1
  • 8
  • 16
  • 1
    This is better than the accepted answer: it allows multiple filebeat inputs in logstash. In such cases, the "type" property is set to "log" and cannot be modified. – Régis B. Mar 20 '18 at 16:07
  • But isn't this overwriting tags with the last tag (bbb)? And then again, in the if statememt, if the tags were an array or single string, then both IF's would work. So logically this is incorrect, but maybe logstash has a different logic inside if's – meso_2600 Oct 28 '18 at 18:06
0

I think logstash can't read more than 2 files in Input section . try the below 

input {
    file {
            type => "technical"
            path => "/home/technical/log"
    }
    file {
            type => "business"
            path => "/home/business/log"
    }
 file {
            type => "business1"
            path => "/home/business/log1"
    }
}
Osama AbuSitta
  • 3,918
  • 4
  • 35
  • 51
KM Prak
  • 127
  • 1
  • 1
  • 6