Azure Function Poison Queue Alerting

Question

Is it possible to create alerts off of posion queue? I have a blob trigger function but when it fails x number of times, it throws this message onto the poison queue. What I am trying to do is create an alert when that poison queue has a count of x.

1. I can do this with a sidecart approach, where I have an independant service that monitors poison queues
2. Saw this approach https://stackoverflow.com/a/46334184/1134076
3. Was hoping to use the blobproperties override to store some metadata so I can keep track of failures but there doesn't seem to be a way to do this? I was thinking on on the final attempt track an event to state moving message to poison queue

Is there a better way of doing this?

        [FunctionName("MyFunction")]
        public async Task Run(
            [BlobTrigger("input-queue", Connection = "ConnectionString")] Stream blobContent,
            string name,
            System.Uri uri,
            IDictionary<string, string> metaData,
            BlobProperties properties,
            ILogger log)

EDIT Noticed this overload for function:

        [FunctionName("MyFunction")]
        public async Task Run(
            [BlobTrigger("input-queue", Connection = "ConnectionString")] ICloudBlob blobContent,
            string name,
            System.Uri uri,
            IDictionary<string, string> metaData,
            BlobProperties properties,
            ILogger log)
        {
            if (!blobContent.Metadata.ContainsKey("mycount"))
            {
                blobContent.Metadata["mycount"] = "1";
                await blobContent.SetMetadataAsync();
            }
            else
            {
                var value = int.Parse(blobContent.Metadata["mycount"]);
                value++;
                blobContent.Metadata["mycount"] = value.ToString();
                await blobContent.SetMetadataAsync();
            }

            throw new Exception(("Testing Function"));
        }

Seems overkill but I can track failure count here and write logic to track custom events.

Answer 1

The approach I prefer is to have an Azure Function that periodically counts the number of items in the poison queues and publishes that as an Application Insights metric. Then you can set up queries/dashboards/alerting from there.

The basic overview of this approach is here. Note that modern Azure Functions lets you use DI and other niceties.

Example of an every-5-minutes timer-triggered function that gets poison queue counts and publishes them as a metric:

[FunctionName(nameof(TrackQueueMetrics))]
public async Task TrackQueueMetrics([TimerTrigger("0 */5 * * * *")] TimerInfo message)
{
  var queues = await _queueService.GetAllQueuesAsync();
  var poisonQueues = queues.Where(x => x.EndsWith("-poison", StringComparison.InvariantCultureIgnoreCase)).ToList();
  var poisonQueueCounts = await Task.WhenAll(poisonQueues.Select(_queueService.GetApproximateMessagesCountAsync));
  var fatalErrors = poisonQueueCounts.Sum();
  _metrics.TrackFatalErrorsCount(fatalErrors);
}

GetAllQueuesAsync() is essentially:

public async Task<IReadOnlyList<string>> GetAllQueuesAsync()
{
  var result = new List<string>();
  await foreach (var item in _queueServiceClient.GetQueuesAsync())
      result.Add(item.Name);
  return result;
}

GetApproximateMessagesCount() is essentially:

public async Task<int> GetApproximateMessagesCountAsync(string queueName)
{
  var properties = await _queueServiceClient.GetQueueClient(queueName).GetPropertiesAsync();
  return properties.Value.ApproximateMessagesCount;
}

and TrackFatalErrorsCount is essentially:

public sealed class SingletonMetricsClient
{
  private readonly Metric _fatalErrorsCountMetric;

  public SingletonMetricsClient(TelemetryConfiguration telemetryConfiguration)
  {
    var client = new TelemetryClient(telemetryConfiguration);
    _fatalErrorsCountMetric = client.GetMetric("FatalErrors");
  }

  public void TrackFatalErrorsCount(int count) => _fatalErrorsCountMetric.TrackValue(count);
}

Once you have it as a metric, then you can query it, build a graph for your Azure dashboard, and/or set up Application Insights alerting.