3

I want an Azure Alert to trigger when a certain function app fails. I set it up as a GTE 1 threshold on the [function name] Failed metric thinking that would yield the expected result. However, when it runs daily I am getting notifications that the alert fired but I cannot find anything in the Application Insights to indicate the failure and it appears to be running successfully and completing.

Here is the triggered alert summary:

enter image description here

Here is the invocation monitoring from the portal showing that same function over the past few days with no failures:

enter image description here

And here is an application insights search over that time period showing no exceptions and all successful dependency actions:

enter image description here

The question is - what could be causing a Azure Function Failed metric to be registering non-zero values without any telemetry in Application Insights?

Update - here is the alert configuration

enter image description here

And the specific condition settings-

enter image description here

Failures blade for wider time range:

enter image description here

There are some dependency failures on a blob 404 but I think that is from a different function that explicitly checks for the existence of blobs at paths to know which files to download from an external source. Also the timestamps don't fall in the sample period.

enter image description here

No exceptions:

enter image description here

Gavin H
  • 10,274
  • 3
  • 35
  • 42
  • 1
    Can you share your alert configuration details/screenshot please. – krishg Sep 14 '20 at 18:21
  • @KrishnenduGhosh-MSFT Sure! I have updated the description with a screenshot of the alert configuration. Thanks – Gavin H Sep 14 '20 at 18:26
  • Thanks. Would be great if you click that condition and share the screenshot. – krishg Sep 14 '20 at 18:30
  • @KrishnenduGhosh-MSFT updated again - thanks. – Gavin H Sep 14 '20 at 18:34
  • Wondering where is the filter condition on the failures only from your 'particular' function in this alert config. I suspect this alert is getting triggered on any failure logged into this application insights instance (I mean includes any other apps/functions also). You can go to your App Insights->"**Failure**" blade to find out if it's indeed the case ![](https://pasteboard.co/Jr59AFs.png). Yes, I saw you shared one screenshot in your concerned time period, but want to be sure it's not an oops of UTC v/s your local time by seeing raw failures :). – krishg Sep 14 '20 at 18:58
  • @KrishnenduGhosh-MSFT the particulr function in the alert is in the blurred out text there before 'Failures'. When creating the alert I saw one for each function in the function app so I created alerts for each one separately. I don't think it is timezone as the function gets triggered daily at 6pm local (pacific) which is when alert shows up and log shows it ran. Here is the failures blade: – Gavin H Sep 14 '20 at 19:22
  • @KrishnenduGhosh-MSFT I put the screenshots of the failure blade tabs in the question body – Gavin H Sep 14 '20 at 19:26
  • 1
    Thanks. I am checking. – krishg Sep 14 '20 at 19:47
  • Ok. I still think the alert is getting fired irrespective of the source of failure (those blob ones). And I am still not clear how you are separating alert for each function (like any custom `where` condition) https://learn.microsoft.com/azure/azure-monitor/platform/alerts-log – krishg Sep 14 '20 at 21:53
  • @KrishnenduGhosh-MSFT it isn't a log alert, it is a metric alert. When I went to create it there are a handful of built in metrics available for each function in my function app. I selected the one the was "[function name] Failures" to use as the metric. https://pasteboard.co/Jr6kghT.png – Gavin H Sep 14 '20 at 22:02
  • @KrishnenduGhosh-MSFT and I double checked and don't see how any of those dependency failures were during the time period when this alert triggered. I created this side by side showing the alert trigger information next to a log search for dependency failures and double checked the timezones and you can see that the dependency failures are all around 5AM local time (when a different function does it's processing) while the alert triggered around 6PM local: https://pasteboard.co/Jr6rhNI.png – Gavin H Sep 14 '20 at 22:15
  • Thanks. Let me take a deeper look. – krishg Sep 14 '20 at 22:20
  • I tried to reproduce your scenario, but unfortunately works fine as expected for me. I suggest you to post this in [Microsoft Q&A](https://learn.microsoft.com/en-us/answers/index.html) to get a closer troubleshooting from product SMEs. – krishg Sep 15 '20 at 06:36
  • 2
    @GavinH, not sure what's happening in your side as it's ok at my side. But as a workaround, you can use the `custom log search` to define the alert which is more trustable. – Ivan Glasenberg Sep 15 '20 at 06:52

1 Answers1

6

Per comment on the question by @ivan-yang I have switched the alerting to use a custom log search instead of the built-in Azure Function metric. At this point that metric seems to be pretty opaque as to what is triggering it and it was triggering every day when I ran the Azure Function with no apparent underlying failure. I plan to avoid this metric now.

My log based alert is using the following query for now to get what I was looking for (an exception happened or a function failed):

requests
| where success == false
| union (exceptions)
| order by timestamp desc

Thanks to @ivan-yang and @krishnendughosh-msft for the help

Gavin H
  • 10,274
  • 3
  • 35
  • 42
  • How do you turn this query into an alert rule? I am trying to use it for my Azure Function but there is no custom choice available under "Select a signal". `custom log search` is not available in the list of signals. – Zizzipupp Dec 09 '21 at 12:09
  • 1
    I got it, I had to use Application Insights as target resource instead of the function app to access the `custom log search` – Zizzipupp Dec 09 '21 at 12:51