Sensu: best practices to only handle a check after n occurences

Question

I've recently upgraded from Sensu 0.24 to 1.2 and noticed that checks where triggered immediately to the referenced handlers.

On the old checks (with v0.24), checks had an "occurences" attribute to filter out noise. I only want checks to be handled by handlers on every n occurences, i.e. a http check must fail 5 times before the pagerduty handler will be triggered. This behaviour seems to have changed with the sensu upgrade.

As I understand, a handler is supposed to include a filter to sort out events based on attributes. So suppose this is my check:

 {
  "checks": {
    "examplecom_http": {
      "command": "check-http.rb --url https://example.com -s -k -q 'Keyword'",
      "handlers": ["default","pagerduty"],
      "subscribers": ["network"],
      "interval": 60,
      "occurrences": 5
    }
  }
}

In previous versions (or at least that was my understanding), this check would only be handled after 5 minutes (5 occurences for 60 second intervals) of failure. This doesn't work anymore, so now the handler should include a filter to handle occurences:

{
  "handlers": {
    "pagerduty": {
      "type": "pipe",
      "command": "/etc/sensu/plugins/pagerduty.rb",
      "severities": ["critical"],
      "filter": "occurences"
    }
  }

And the "occurences" filter would look like this:

{
  "filters": {
    "occurences": {
      "attributes": {
        "occurrences": "eval: value >= 5"
      }
    }
  }
}

However, whatever comes after the eval part, be it value >= 5 or value < 5, the effect is the same and the pagerduty handler gets executed. I've tried using the negate directive with true and false but it seems my understanding of how filtering and occurences work for checks is just not correct. Maybe checks don't count their occurences at all?

Can somebody help and explain this?

score 3 · Answer 1 · answered Oct 30 '18 at 05:34

As it stands right now, it looks like occurrences in your handlers JSON block is spelled incorrectly as occurences.

You will want "negate": false the way that your eval is currently written. When negate is false, this means that the handler will only trigger when the filter evaluation is true - negate will simply reverse the result so handlers trigger when the filter evaluation is false.

Finally, if you are looking to trigger the handler "every n occurrences" instead of every occurrence after the 5th, consider this evaluation:

{
  "filters": {
    "every_five": {
      "attributes": {
        "occurrences": "eval: value % 5 == 0"
      }
    }
  }
}

This way, when the number of occurrences is divisible by 5, the handler will trigger, but you will need to add the every_five filter to your handler definition. You can change filter to filters and pass an array instead, like so:

{
  "handlers": {
    "pagerduty": {
      "type": "pipe",
      "command": "/etc/sensu/plugins/pagerduty.rb",
      "severities": ["critical"],
      "filters": [
        "occurrences",
        "every_five"
    }
  }
}

This will allow you to use the built-in filter "occurrences" as well as your custom filter.

Sensu: best practices to only handle a check after n occurences

1 Answers1