How to send Cloudwatch log details via email?

Question

The diagram below is what I am trying to achieve. In brief, to send CloudTrail logs to CloudWatch log group then scan it for certain events and finally send email alerts if there is an concerting event.

I am following this official documentation which also has a sample CloudFormation templates: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-cloudformation-template-to-create-cloudwatch-alarms.html

Using the CloudFormation templates above, I have been able to send the email alerts. However the alerts are very basic; it does not send key information like which user initiated this event, when did it occur etc.

Logically thinking AWS::Logs::MetricFilter should pass the value to AWS::CloudWatch::Alarm which would then send the information. I have looked at the documentation of both MetricFilter and Alarm services. Dimension comes closer to what I want but not yet able to read the information from logs.

I would have thought this is a common use case and there would be documentation. Am I missing something glaringly obvious here? Has anyone here solved this issue?

AWS::Logs::MetricFilter block:

"AuthorizationFailuresMetricFilter": {
    "Type": "AWS::Logs::MetricFilter",
    "Properties": {
        "LogGroupName": { "Ref" : "LogGroupName" },
        "FilterPattern": "{ ($.errorCode = \"*UnauthorizedOperation\") || ($.errorCode = \"AccessDenied*\") }",
        "MetricTransformations": [
            {
                "MetricNamespace": "CloudTrailMetrics",
                "MetricName": "AuthorizationFailureCount",
                "MetricValue": "1"
            }
        ]
    }
},

AWS::CloudWatch::Alarm block

  "AuthorizationFailuresAlarm": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
          "AlarmName" : "CloudTrailAuthorizationFailures",
          "AlarmDescription" : "Alarms when an unauthorized API call is made.",
          "AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }],
          "Dimensions": [
             {
                "Name": "errorCode",
                "Value": ""
             },
             {
                "Name": "userIdentity",
                "Value": ""
             }
          ],
          "MetricName" : "AuthorizationFailureCount",
          "Namespace" : "CloudTrailMetrics",
          "ComparisonOperator" : "GreaterThanOrEqualToThreshold",
          "EvaluationPeriods" : "1",
          "Period" : "300",
          "Statistic" : "Sum",
          "Threshold" : "1"

      }
  },

You can't do this. It's a missing feature of CloudWatch Logs that Amazon really needs to add. — Mark B, Aug 16 '17 at 13:42
One would have thought this is one of the most common use case. Odd, its not implemented yet. — Sushan Ghimire, Aug 16 '17 at 13:58

score 6 · Accepted Answer · answered Aug 16 '17 at 17:49

6

This is not possible.

Amazon CloudWatch Logs will accept information from AWS CloudTrail and, upon finding messages that match a pre-defined filter, will increment a metric count.

An Amazon CloudWatch alarm can then be triggered when the metric exceeds a certain threshold. However, there is no direct connection between the incoming data that generated the metrics and the alarm that triggers based upon the threshold.

Think of it like a turnstile counting people who enter a subway. The turnstile counts the number of people, but does not retain information about the people who passed through. In the same way, the CloudWatch alarm counts the events but does not have any information about the events that were counted.

answered Aug 16 '17 at 17:49

John Rotenstein

241,921
22
380
470

Thanks for the answer John. I really want to sniff CloudTrail logs and send out alerts to incident response teams with all details possible. How would you solution this yourself? – Sushan Ghimire Aug 17 '17 at 15:09
Are you wanting to notify on one specific event, or only when a calculated threshold has been passed (eg more than 5 occurrences of the event over a 1 hour period)? The former should be easy, the latter is hard because the data is spread across many events over time. – John Rotenstein Aug 17 '17 at 16:37
I want to send an alert when an event happens not necessarily calculated threshold. But what I do need is the alert to be detailed and helpful for an investigation. – Sushan Ghimire Aug 24 '17 at 20:24
If you just want to be notified when a specific event happens, then configure this with Amazon CloudWatch Events. You can define the API call or event as the trigger, and an SNS topic as the destination. – John Rotenstein Aug 24 '17 at 20:43

How to send Cloudwatch log details via email?

1 Answers1