1

Currently my logstash input is listening to filebeat on port XXXX,my requirement is to collect log data only from a particular hosts(let's say only from Webservers). I dont want to modify the filebeat configuration directly on to the servers but I want allow only the webservers logs to listen in.

Could anyone suggest how to configure the logstash in this scenario? Following is mylogstash input configuration.

**input {
  beats {
    port => 50XX
  }
}**
Prashanth
  • 23
  • 4

2 Answers2

1

In a word, "no", you cannot configure the input to restrict which hosts it will accept input from. What you can do is drop events from hosts you are not interested in. If the set of hosts you want to accept input from is small then you could do this using a conditional

if [beat][hostname] not in [ "hosta", "hostb", "hostc" ] { drop {} }

Similarly, if your hostnames follow a fixed pattern you might be able to do it using a regexp

if [beat][hostname] !~ /web\d+$/ { drop {} }

would drop events from any host whose name did not end in web followed by a number.

If you have a large set of hosts you could use a translate filter to determine if they are in the set. For example, if you create a csv file with a list of hosts

hosta,1
hostb,1
hostc,1

then do a lookup using

translate {
    field => "[beat][hostname]"
    dictionary_path => "/some/path/foo.csv"
    destination => "[@metadata][field]"
    fallback => "dropMe"
}
if [@metadata][field] == "dropMe" { drop {} }
Badger
  • 3,943
  • 2
  • 6
  • 17
0

@Badger - Thank you for your response! As you rightly mentioned, I have large number of hosts, and all my web servers follows naming convention(for an example xxxwebxxx). Could you please brief me the following

translate {
    field => "[beat][hostname]"
    dictionary_path => "/some/path/foo.csv"
    destination => "[@metadata][field]"
    fallback => "dropMe"
}
if [@metadata][field] == "dropMe" { drop {}

Also, please suggest how to add the above to my logstash.conf, PFB this is how my logstash.conf looks like

input {
  beats {
    port => 5xxxx
  }
}

filter {
  if [type] == "XXX" {

    grok {
      match => [ "message", '"%{TIMESTAMP_ISO8601:logdate}"\t%{GREEDYDATA}']
    }

    grok {
      match => [ "message", 'AUTHENTICATION-(?<xxx_status_code>[0-9]{3})']
    }

    grok {
      match => [ "message", 'id=(?<user_id>%{DATA}),']
    }

    if ([user_id] =~ "_agent") {
      drop {}
    }

    grok {
      match => [ "message", '%{IP:clientip}' ]
    }

    date {
      match => [ "logdate", "ISO8601", "YYYY-MM-dd HH:mm:ss"]
      locale => "en"
    }

    geoip {
      source => "clientip"
    }
  }
}
output {
  elasticsearch {
    hosts => ["hostname:port"]
  }

  stdout { }
}
Prashanth
  • 23
  • 4