45

I have alarms set up to tell me when my load balancers are throwing 5xxs using the HTTPCode_Backend_5XX metric with the sum statistic. The issue is that sum registers 0 as no data points, so when no 5xxs are thrown, the alarm is treated as insufficient data. This is especially frustrating, because I have SNS setup to notify me whenever we get too many 5xxs (alarm state) and whenever things go back to normal. Annoyingly, 0 5xxs means we're in INSUFFICIENT DATA status, but 1 5xx means we're in OK status, so 1 5xx triggers everyone getting notified that stuff is OK. Is there any way around this? Ideally, I'd like to just have 0 of anything show up as a zero data point instead of no data at all (insufficient data).

Eli
  • 36,793
  • 40
  • 144
  • 207
  • Do you have much traffic on the ELB? No requests at all "should" be what causes insufficient data on the data points, as opposed to some traffic but 0 errors. – Michael - sqlbot Nov 10 '15 at 23:10
  • Yes. There's lots of traffic. Cloudwatch checks the metric every five minutes, and I have thousands of requests during that time. – Eli Nov 10 '15 at 23:12

3 Answers3

38

As of March 2017, you can treat missing data as acceptable. This will prevent the alarm from being marked as INSUFFICIENT.

CloudWatch missing data option

You can also set this in CloudFormation using the TreatMissingData property. For example:

TreatMissingData: notBreaching
Jonathan
  • 13,947
  • 17
  • 94
  • 123
6

We had a similar issue for some of our alarms. You can actually avoid this behaviour with some work, if you really want to deal with the overhead.

What we have done is, instead of sending SNS notifications directly to e-mails, we have created a lambda function and triggered it once we have the notification in the SNS topic.

This way, you will have more control over the actions you can take once the alarms are triggered. As the context will provide you old state value as well.

The good news is, there is already a lambda template to get started. https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/

Just pick the one that is designed to send cloudwatch alarms to slack. You can then modify the code as you wish, either dismiss the slack part and just use emails, or keep it with slack. (which is what we did and it works like a charm)

Muhammet Can
  • 1,304
  • 2
  • 16
  • 30
2

I asked for this in the AWS forums two years ago :-( https://forums.aws.amazon.com/thread.jspa?threadID=153753&tstart=0

Unfortunately you cannot create notifications based on specific state changes (in your case you want a notification when state changes from ALARM to OK, but not when state changes from INSUFFICIENT to OK). I can only suggest that you also ask for it and hopefully it will eventually be added.

For metrics that are often in the INSUFFICIENT state I generally just create notifications for ALARMS and I don't have notifications on OK for these metrics - if I want to confirm that things are OK I use the AWS mobile app to check on things and see if they have resolved.

SeanN
  • 633
  • 8
  • 11