9

We want to set up a server for logstash for a couple of different project in our company. Now I try to enable them in Kibana. My question is: If I have different patterns of the logfiles, how can I build for them a filter? example: logstash.conf:

input {
  file {
    type => "A"
    path => "/home/logstash/A/*"
    start_position => "beginning"
  }
 file {
    type => "B"
    path => "/home/logstash/B*"
    start_position => "beginning"
  }
}

filter {
  multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
  }
  grok {
     type => A
     match => [ "message", "%{TIMESTAMP_ISO8601:logdate} %{DATA:thread %{LOGLEVEL:level}\s*%{DATA:logger_name}\s*-\s*%{GREEDYDATA:log_text}"]

    add_tag => [ "level_%{level}" ]
  }
  date {
        match => ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"]
  }
  grok {
        type => B
        match => [ any other pattern ... 
 }
}
output {
  elasticsearch { embedded => true }
}

do I have to create for each project (A,B,C,...) an own filter, and what do I have to do, when I have for each project again different pattern of the logfiles?

baudsp
  • 4,076
  • 1
  • 17
  • 35
user3300940
  • 91
  • 1
  • 1
  • 3
  • thanks Ben, I will try it. And what shall I do, when I have different logfile patterns wihtin A and B, when A and B are the projects? – user3300940 Feb 12 '14 at 10:49

1 Answers1

11

You only need to create a filter for all projects.

For Logstash 1.3.3, You can use if statement to distinct each project grok. For example, 

filter {

   multiline {
       pattern => "^%{TIMESTAMP_ISO8601}"
       negate => true
       what => "previous"
   }

   if [type] == "A"  {
      grok {
          match => [ any other pattern ... 
      }
   }
   else if [type] == "B" {
      grok {
          match => [ any other pattern ... 
      }
   }
}

Hope this can help you.

Ban-Chuan Lim
  • 7,840
  • 4
  • 35
  • 52
  • thanks Ben, I will try it. And what shall I do, when I have different logfile patterns wihtin A and B, when A and B are the projects? – user3300940 Feb 12 '14 at 10:50
  • One input(file) will have one type. So, if you have different logfile patterns in A, you need to separate the log first. – Ban-Chuan Lim Feb 12 '14 at 13:28
  • Thanks Ben.Now for my understanding: When I have two different Webservices (A and B) with different patterns of the logfiles, then it is not possible to display them in one Kibana GUI installation. I would need for every Webservices a own Kibana GUI, right? It is not possible in one GUI with two Dashboards? – user3300940 Feb 13 '14 at 15:28
  • 1
    I think you have misunderstanding. If you want to use grok to parse your logfiles in Logstash, the log pattern is prefer to same, otherwise your grok pattern will be complicated. Two or more web services log files can save to one elasticsearch(Search Engine). You can have one Kibana GUI with two different dashboard. For example, you can configure one dashboard with type:A and another with type:B – Ban-Chuan Lim Feb 14 '14 at 01:12