ALerting in Riemann?

Question

I am using ELK (logstash, ES, Kibana) stack for log analysis and Riemann for alerting. I have logs in which users is one of the fields parsed by logstash and I send the events to riemann from riemann output plugin.

Logstash parses logs and user is one of the field. Eg: logs parsed

Timestamp              user     command-name
 2014-06-07...         root      sh ./scripts/abc.sh
 2014-06-08...         sid       sh ./scripts/xyz.sh
 2014-06-08...         abc       sh ./scripts/xyz.sh
 2014-06-09...         root      sh ./scripts/xyz.sh

Logstash:

riemann {
    riemann_event => {
        "service"     => "logins"
        "unique_user" => "%{user}"
    }
}

So users values will be like: root, sid, abc, root, sid, def, etc....

So I split stream by user i.e one stream for each unique user. Now, I want to alert when number of unique users count go more than 3. I wrote the following but it's not achieving my purpose.

Riemann:

(streams

 (where (service "logins")
  (by :unique_user
    (moving-time-window 3600 
     (smap (fn [events]
      (let
        [users (count events)]
         (if (> users 3)
          (email "abc@gmail.com")       
     ))))))))

I am new to Riemann and clojure. Any help is appreciated.

score 1 · Answer 1 · answered Jan 16 '17 at 20:39

email returns a stream. Therefore, for it to work, you must either use it as a stream, by passing it as a parameter to another stream, or use call-rescue to send an event to it directly. Additionally, streams that are meant to receive events from multiple sources (such as your alert destination) should be created once, and stored in a variable for re-use.

First approach, using only abstract streams:

(let [alert (email "abc@gmail.com")]
  (streams
    (where (service "logins")
      (by :unique_user
        (moving-time-window 3600
          (smap folds/count
            (where (> metric 3) alert)))))))

Second approach, using call-rescue:

(let [alert (email "abc@gmail.com")]
  (streams
    (where (service "logins")
      (by :unique_user
        (moving-time-window 3600
          (fn [events]
            (when (> (count events) 3)
              (call-rescue (last events) alert))))))))

ALerting in Riemann?

1 Answers1