2

So I've created a logging alert policy on google cloud that monitors the project's logs and sends an alert if it finds a log that matches a certain query. This is all good and fine, but whenever it does send an email alert, it's barebones. I am unable to include anything useful in the email alert such as the actual message, the user must instead click on "View incident" and go to the specified timeframe of when the alert happened.

Is there no way to include the message? As far as I can tell viewing the gcp Using Markdown and variables in documentation templates doc on this.

I'm only really able to use ${resource.label.x} which isn't really all that useful because it already includes most of that stuff by default in the alert.

Could I have something like ${jsonPayload.message}? It didn't work when I tried it.

DazWilkin
  • 32,823
  • 5
  • 47
  • 88
Dmytro Lysak
  • 397
  • 1
  • 4
  • 11

3 Answers3

5

Probably (!) not.

To be clear, the alerting policies track metrics (not logs) and you've created a log-based metric that you're using as the basis for an alert.

There's information loss between the underlying log (that contains e.g. jsonPayload) and the metric that's produced from it (which probably does not). You can create Log-based metrics labels using expressions that include the underlying log entry fields.

However, per the example in Google's docs, you'd want to consider a limited (enum) type for these values (e.g. HTTP status although that may be too broad too) rather than a potentially infinite jsonPayload.

DazWilkin
  • 32,823
  • 5
  • 47
  • 88
0

It is possible. Suppose you need to pass "jsonPayload.message" present in your GCP log to documentation section in your policy. You need to use "label_extractor" feature to extract your log message.

I will share a policy creation JSON file template wherein you can pass "jsonPayload.message" in the documentation section in your policy.

policy_json = {
  "display_name": "<policy_name>",
  "documentation": {
    "content": "I have the extracted the log message:${log.extracted_label.msg}",
    "mime_type": "text/markdown"
  },
  "user_labels": {},
  "conditions": [
    {
      "display_name": "<condition_name>",
      "condition_matched_log": {
        "filter": "<filter_condition>",
        "label_extractors": {
          "msg": "EXTRACT(jsonPayload.message)"
        }
      }
    }
  ],
  "alert_strategy": {
    "notification_rate_limit": {
      "period": "300s"
    },
    "auto_close": "604800s"
  },
  "combiner": "OR",
  "enabled": True,
  "notification_channels": [
    "<notification_channel>"
  ]
}
Stephen Ostermiller
  • 23,933
  • 14
  • 88
  • 109
Naveen Thomas
  • 1
0

It is possible, but hard to understand from Google documentation. The answer is a modification of the answer provided by Naveen Thomas, depending on the log entry you are trying to pass.

For this, you will have to open the log entry in Logs Explorer, and take a look at the fields available.

For example, in Cloud SQL error log, the field which contains the message is "textPayload". To pass this to the notification, you would create an log-based alert policy and use the "Extract log labels" option (label_extractors in JSON), enter a display name (I used "msg") and enter "textPayload" as "log field name".

Then add $"{log.extracted_label.msg}" to the "Documentation" field, formatting as you see fit.

In JSON, my policy looks like this:

{
  "name": "projects/<project_name>/alertPolicies/<Policy_ID>",
  "displayName": "<Policy name>",
  "documentation": {
    "content": "CloudSQL Instance Log-based alert in project <project_name> detected:\n\n${log.extracted_label.msg}",
    "mimeType": "text/markdown"
},
"userLabels": {},
"conditions": [
{
  "name": "projects/<project_name>/alertPolicies/<Policy_ID>/conditions/<Condition_ID?>",
  "displayName": "Log match condition",
  "conditionMatchedLog": {
    "filter": "resource.type=\"cloudsql_database\"\nresource.labels.database_id=\"<project_name>:<instance_ID>\"\nlogName=\"projects/<project_name>/logs/cloudsql.googleapis.com %2Fsqlserver.err\"\nseverity=(INFO OR ERROR OR CRITICAL OR ALERT OR EMERGENCY)",
    "labelExtractors": {
     "msg": "EXTRACT(textPayload)"
      }
    }
  }
],
  "alertStrategy": {
    "notificationRateLimit": {
      "period": "300s"
    },
    "autoClose": "604800s"
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": [
    "projects/<project_name>/notificationChannels/<notificationChannel_ID>"
  ],
  "creationRecord": {
    "mutateTime": "2023-01-01T07:11:53.406233445Z",
    "mutatedBy": "<User>"
  },
  "mutationRecord": {
    "mutateTime": "2023-01-01T13:22:19.917589988Z",
    "mutatedBy": "<User>"
  }
}

References:

https://cloud.google.com/logging/docs/logs-based-metrics/labels https://cloud.google.com/monitoring/alerts/doc-variables#doc-vars

Jamaluus
  • 1
  • 2