0

I want to collect and process logs from dnsmasq and I´ve decided to use ELK. Dnsmasq is used as a DHCP Server and as a DNS Resolver and hence it creates log entries for both services.

My goal is to send to Elasticsearch all DNS Queries with the requester IP, requester hostname (if available) and requester mac address. That will allow me to group the request per mac address regardless if the device IP changed or not, and display the host name.

What I would like to do is the following:

1) Read the entries like:

Mar 30 21:55:34 dnsmasq-dhcp[346]: 3806132383 DHCPACK(eth0)  192.168.0.80 04:0c:ce:d1:af:18 air

2) Store temporarily the relationship:

192.168.0.80 => 04:0c:ce:d1:af:18

192.168.0.80 => air

3) Enrich the entries like the one below adding the mac address and hostname. If the hostname was empty I would add the mac address.

Mar 30 22:13:05 dnsmasq[346]: query[A] imap.gmail.com from 192.168.0.80

I found a module called “memorize” that would allow me to store them but unfortunately does not work with the latest version of Logstash

The versions I´m using:

ElastiSearch 2.3.0
Kibana 4.4.2
Logstash 2.2.2

And the logstash filter (this is my first attempt with logstash and hence I´m sure the configuration file can be improved)

input {
  file {
    path => "/var/log/dnsmasq.log"
    start_position => "beginning"
    type => "dnsmasq"
  }
}  

filter {
  if [type] == "dnsmasq" {
    grok {
      match =>  [ "message", "%{SYSLOGTIMESTAMP:reqtimestamp} %{USER:program}\[%{NONNEGINT:pid}\]\: ?(%{NONNEGINT:num} )?%{NOTSPACE:action} %{IP:clientip} %{MAC:clientmac} ?(%{HOSTNAME:clientname})?"]
      match =>  [ "message", "%{SYSLOGTIMESTAMP:reqtimestamp} %{USER:program}\[%{NONNEGINT:pid}\]\: ?(%{NONNEGINT:num} )?%{USER:action}?(\[%{USER:subaction}\])? %{NOTSPACE:domain} %{NOTSPACE:function} %{IP:clientip}"]
      match =>  [ "message", "%{SYSLOGTIMESTAMP:reqtimestamp} %{USER:program}\[%{NONNEGINT:pid}\]\: %{NOTSPACE:action} %{DATA:data}"]
    }

    if [action] =~ "DHCPACK" {

    }else if [action] == "query" {

    }else
    {
      drop{}
    }
  }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Questions:

1) Is there an alternative to the plugin “memorize” working with the latest logstash version? Either another plugin or different procedure.

2) Shall I downgrade logstash to a version before 2 (I think the previous is 1.5.4)? If so, is there any known sever issue or incompatibility with elasticsearch 2.2.1?

3) Or shall I modify the plugin “memorize” allowing logstash 2.x (if so I´ll appreciate any pointer on how to start)?

Jofre
  • 215
  • 1
  • 16
  • It seems memorize filter just keeps the last value of an event. Will that be useful to you? – alpert May 19 '16 at 14:54
  • Memorize would be perfect but doesn't work for logstash 2.x – Jofre May 19 '16 at 15:06
  • By changing the dependency number you can install memorize. Or just install it from:https: //github.com/jofrep/logstash-filter-memorize. I tried and it seem to work. – alpert May 19 '16 at 15:26
  • This was my try. Is my repo. But I failed to install it. It will be good to know how you did it – Jofre May 19 '16 at 15:43
  • Kinda funny :) I answer it with the way I made it work. Hope helps. – alpert May 19 '16 at 15:46

2 Answers2

3

There's no need to repack the memorize plugin for this in my opinion. You can use the aggregate filter to achieve what you want. 

...

# record host/mac in temporary map
if [action] =~ "DHCPACK" {
  aggregate {
     task_id => "%{clientip}"
     code => "map['clientmac'] = event['clientmac']; map['clientname'] = event['clientname'];"
     map_action => "create_or_update"
     # timeout set to 48h
     timeout => 172800
  }
}

# add host/mac where/when needed
else if [action] == "query" {
   aggregate {
     task_id => "%{clientip}"
     code => "event['clientmac'] = map['clientmac']; event['clientname'] = map['clientname']"
     map_action => "update"
   }
}
Jofre
  • 215
  • 1
  • 16
Val
  • 207,596
  • 13
  • 358
  • 360
  • I used the "aggregate" plugin with some modification to solve typos and ensure the mapping was permanent. Nevertheless although I set the timeout to 0 and use a single thread often the mapping are lost and the log entries are not populated with the mapped values (until a new DHCPACK is seen). Any suggestion? – Jofre May 24 '16 at 18:30
  • Sorry for the typos. Can you try to add `aggregate_maps_path => /path/to/maps` and monitor that file to see how the mapping are looking like over time? – Val May 25 '16 at 03:53
  • Looking at the results I see that the maps last for 30 min after a DHCPACK. In some places it says that the maps are removed after 1800 seconds (30 min), in other it says that by default these are never removed. In any case I set it to "0" (never remove). – Jofre May 25 '16 at 08:23
  • So with `timeout=>0`, the map is never evicted, right? Can you start logstash with `--debug` so that we get more insights into what logstash is doing? – Val May 25 '16 at 08:25
  • I could not use aggregate_maps_path as it is failling. I created a kibana report looking at the DHCP calls and at the variable `clientname` and looks to me that the `timeout=>0` is ignored. After 30min the variable is not populated anymore. I'll check with the `--debug` – Jofre May 25 '16 at 08:35
  • my bad, `aggregate_maps_path` is not available in Logstash 2.2.2. Any chance to upgrade to 2.3.2 and see if it makes a difference? – Val May 25 '16 at 08:37
  • Upgraded to 2.3.2 but still unable to use 'aggregate_maps_path'. In any case I found the problem. If you set the 'timeout = 0" in 'aggregate', regardless of what the documentation says, the timeout is not set to "not expire" but to the default 1800 second. I set the timeout to 48h and at the moment it looks good. – Jofre May 25 '16 at 16:12
0

So to use memorize with logstash >2.0

  • Clone the repository.
  • Open file logstash-filter-memorize.gemspec
  • Change s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 2.0.0' as s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 3.0.0'
  • Build plugin via: gem build logstash-filter-memorize.gemspec
  • Install it via: $ bin/logstash-plugin install /path/to/memorize/logstash-filter-memorize-0.9.1.gem

I tried it and seems to work.

alpert
  • 4,500
  • 1
  • 17
  • 27