6

I want grafana to report how many messages are in a dead letter queue at any given time. The messages that end up within this queue are evaluated and parsed due to the nature of them ending up in this queue.

I've configured grafana to read NumberOfMessagesSent from a dead-letter queue, however, the value is always 0, because; I think messages that end up here are sent from another queue (via SQS).

Even though I can see messages have been sent to the dead-letter queue (not programatially), but by another queue that's configured after X number of receives.

grafana config for NumberOfMessagesSent

Is there a solution to this?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
steadweb
  • 15,364
  • 3
  • 33
  • 47

2 Answers2

17

Quick answer:

Only metric which you can really monitor on SQS DLQ is ApproximateNumberOfMessagesVisible.

TLDR:

When new message arrives to SQS DLQ neither NumberOfMessagesSent or NumberOfMessagesReceived is increased. Citation from AWS Documentation:

If you send a message to a dead-letter queue manually, it is captured by the NumberOfMessagesSent metric. However, if a message is sent to a dead-letter queue as a result of a failed processing attempt, it isn't captured by this metric. Thus, it is possible for the values of NumberOfMessagesSent and NumberOfMessagesReceived to be different.

NumberOfMessagesSent means:

How many of messages you have sent to the queue (except messages which arrived to DLQ as result of failure)

NumberOfMessagesReceived means:

How many of messages you have received from the queue

ApproximateNumberOfMessagesVisible means:

Total number or messages, which are visible in the queue. (Remember when you receive message from the queue you have to set visibility timeout == how long message you have received is not visible for others. Also remember you should delete message after successful processing, otherwise it will be after visibility timeout available to others.)

David Navrkal
  • 464
  • 5
  • 8
1

Because the metrics for NumberOfMessagesReceived and NumberOfMessagesSent are dependent on how the message is queued as David Navrkal mentions, I have devised a new solutions for our needs using the metric ApproximateNumberOfMessagesDelayed. On the dlq settings I have added the delay to use this metric.

ie: DelaySeconds: 60 #for alarms

For the yaml see Configure SQS Dead letter Queue to raise a cloud watch alarm on receiving a message.

alex2017
  • 121
  • 1
  • 6