Elastalert simplified multiple rules in one file

Question

I'm writing Elastalart rules for heartbeat i.e if service or machine are/is down, I should get notified. Right now I can create one rule for service per one file like below.

name: My Alert
type: frequency
index: heartbeat-*
num_events: 5
timeframe:
    minutes: 2

filter:
- query:
    query_string:
      query: "url.domain: MY_LOCALHOST01.local AND monitor.status: down"

alert:
- "email"

email:
- "user@example.in"

Is there any way, can I specify multiple rules??... I can specify multiple filter like below

...
filter:
- query: # Filter 1
    query_string:
      query: "url.domain: MY_LOCALHOST01.local AND monitor.status: down"

- query: # Filter 2
    query_string:
      query: "url.domain: MY_LOCALHOST02.local AND monitor.status: down"
...

But Elatalert consider num_events on ALL filters. For example, I dont want to get alert for situation like Filter 1 got 3 hits and Filter 2 got 2 hits i.e 3+2=5 which is equal to num_events.

So, is there any ways the num_events should check per filter? like if Filter 1 got 5 hits and Filter 2 got 3 hits, then I can confirm MY_LOCALHOST01 is really DOWN and send alert.

I don't want multiple files. It would be hard to manage/modify.

Is that ok for you to handle this with the help of separate index? I suspect a way if its OK. — Gibbs, Jun 24 '20 at 19:20
An index to handle the rules logic and check whether enough events occurred to trigger alert. — Gibbs, Jun 25 '20 at 10:06
Are your `filters` based on same set of fields? `domain and status`? — Gibbs, Jun 26 '20 at 07:50
Yes both `"url.domain: MY_LOCALHOST01.local AND monitor.status: down"` are in same `document` — Veerendra K, Jun 26 '20 at 09:08
`Filter1`, `Filter2` works on same set of field `url.domain, monitor.status`. And how many such filters you have. Or do you have filters works on different column combined together in the same rule file? — Gibbs, Jun 26 '20 at 09:13
I have around 10 filters. Filter1, Filter2 works on same set of fields — Veerendra K, Jun 26 '20 at 10:13
What about other filters? I thought of using `change type` with an `index` if they are all on same set of fields. — Gibbs, Jun 26 '20 at 10:34
other filter are also same, just a change of `url.domain` i.e `MY_LOCALHOST03.local`, `MY_LOCALHOST04.local`, `MY_LOCALHOST05.local`, etc — Veerendra K, Jun 26 '20 at 10:46

score 0 · Answer 1 · answered Jun 27 '20 at 11:12

I would suggest you to think before doing this.

To achieve the expected result:

Have rule_type as change instead frequency
Keep the same timeframe.
Monitor on status as you want to check whether it is down
Set filter on monitor field.
Set alert as POST
You can have your own backend API to which you can redirect - You can send the entire document which got changed - Through which you can identify which domain is down. Backend API can write to an index which domain is down. Key name is domain_name. You can keep a counter kind of thing to increase. I am not sure whether we can directly post too ES. But documentation says any end point which accepts JSON.
Now you have your frequencyrule set on the new index. Have your filters as OR - domain1_down : 5 OR domain2_down:5. You can have your same email alerting. But you need to derive which domain from the key or you can have one more field in the index to be used by alerting.

Here the trickiest point is that your config says you want to find 5 downtimes of a domain in 2 minutes of timeframe

By using the aforementioned steps, you can find whether it went down 5 times. But not within 2 minutes time frame. I guess that you can achieve that by keeping a field previous_down_time in the extra index.

It's harder way to achieve what is needed. I don't think there is no other better way than maintaining separate files. That is not harder than this.

I cant accept the answer untill I test it. But I gave you the bounty for your workaround — Veerendra K, Jun 30 '20 at 09:10
I think, there is similar question in github issue. https://github.com/Yelp/elastalert/issues/2849 Do you think, is that related to this — Veerendra K, Jun 30 '20 at 09:17
I think it is related to this - https://elastalert.readthedocs.io/en/latest/ruletypes.html#query-key — Gibbs, Jun 30 '20 at 09:34

Elastalert simplified multiple rules in one file

1 Answers1

Linked