How to implement the retry pattern when processing a message from a queue or topic subscription in Azure Service Bus fails due to transient failure

Question

I haven't found a clear and comprehensive explanation as to how to implement the retry pattern when trying to process a message peeked from an Azure Service Bus. Processing of message is impeded by a transient failure (i.e. a server or service is temporarily unavailable or busy) and message needs to be peeked and its processing tried again after a specific delay.

Answer 1

The Retry pattern can be used to try and process a message again after a transient failure impedes the message processing (i.e. a service or server required is busy). The retry action needs to be delayed to give the systems or services that have the failure time to possibly recover. After a defined number of attempts it should also be possible to just give up and send the message to a Dead Letter Queue for someone to re-enqueue it manually once failure has been corrected.

Possible ways to implement this pattern for messaging processing:

1. Abandoning the message

When configuring the queue or topic subscription, you can set the Message Lock Duration for the delay wanted for retrying, and the Max delivery count to the number of attempts desired. When a transient failure occurs, you can just finish the processing without completing or explicitly abandoning the message. Azure will remove the lock and make the message available again for consumption only after the Message Lock Duration has passed, hence causing the delay expected before the message processing is retried. This approach allows only a constant delay (for exponential backoff see options 2 and 3).

Pros:

• Simplest approach
• Works well for both queues and topics with one or multiple subscriptions (message is returned to the queue, or to the specific subscription).

Cons:

• It does not allow delays with exponential backoff.

2. Use the ScheduledEnqueueTimeUtc property

You can clone the original message and set the ScheduledEnqueueTimeUTC property with a value that represent the delay desired, i.e. current date and time plus the delay (you will need to add a message custom property or a field to the message body to keep an attempt counter, that you can increase after each retry; you can use this counter to calculate the delay as exponential backoff, and also to check the maximum number of attempts after which you can send the message to the Dead Letter Queue). Publish the new message to the same queue/topic and complete (remove) the old one. The message will be available again after the delay has passed. In the case of topics it will work when there is only one subscription (topics with multiple subscription might have the problem of unnecessarily enqueueing a message to other subscriptions that actually succeeded to process the message before).

Pros:

• Allows exponential backoff.

Cons:

• It does not work well for topics with multiple subscriptions (message is send again to all the subscriptions, not just the one that triggered the retry action).

3. Configure a queue specific for the messages to retry

You can clone the original message and invoke the ScheduleMessageAsync method passing the new message and the value in seconds for the delay desired (you will need to add a message custom property or a field to the message body to keep an attempt counter, that you can increase after each retry; you can use this counter to calculate the delay as exponential backoff, and also to check the maximum number of attempts after which you can send the message to the Dead Letter Queue). Publish the new message to the queue specified for the messages to retry, and complete (remove) the old one. Your client will need to subscribe to both the original topic subscription (for the normal processing) and also to the queue you defined for the messages to retry (for retry processing). From the retry queue, the message will be available again after the delay has passed.

Pros:

• Allows exponential backoff.
• It work well for topics with multiple subscriptions (message is send again to all the subscriptions, not just the one that triggered the retry action).

Cons:

• Overkill for queue and topics with just one subscription.