1

I have a Service Bus topic that has 400,000 messages pushed to it from an Azure Function. I have a second Azure Function that receives the messages as a trigger. When the 2nd Function ran, it successfully processed 98% of the messages. It left me with roughly 8,000 messages that failed. Either from an exception, or from my code, the message was abandoned. I've now got 8,000 messages sitting in a subscriber of the topic, that I can't get the Function to re-try processing.

The subscriber was originally set up to only allow 1 delivery of the message. I had done it that way because I was seeing the same message get processed multiple times while debugging. I wasn't sure if that was a side-effect of debugging in Visual Studio locally, or if messages would get sent to a Function multiple times. I assume this is what caused those messages to not have the Function re-run after they were abandoned (if that even is supported?).

I've changed the subscriber delivery count to 5, hoping that it would re-deliver the message to the Function. It didn't. What do I need to do now to get the remaining 8,000 messages to trigger the 2nd Function again?

Worst case is that I can delete the subscription, re-create the subscription and run the first Function over again, which should just publish the data associated with the original missing 8,000 messages (I have logic to handle missing data in the 1st Function). That would cause the 2nd Function to re-fire. I'd like to figure out how to handle it though when the subscription has the orphaned messages like this, as I'll experience this when I promote to production. I want this to be automated and not have to deal with manually cleaning up and re-running the process a 2nd time.

The following shows that there aren't any messages left in the Topic, but they exist still in the subscription. enter image description here

I wrote the Functions in C# Functions 1.0 on Full Framework.

Thomas
  • 24,234
  • 6
  • 81
  • 125
Johnathon Sullinger
  • 7,097
  • 5
  • 37
  • 102
  • you can get the messages from the deadletter or poison queue. – Thomas Apr 04 '18 at 02:43
  • So I cant get the original Service bus triggered Azure Function to run again? I have to write a utility app to go get those messages? – Johnathon Sullinger Apr 04 '18 at 02:48
  • On your azure dashboard, could you see this messages in the deadletter or poison queue ? if so yes you can get them – Thomas Apr 04 '18 at 02:49
  • No, the message count for everything is 0 under the Topic. The subscription has the 8,000 messages but the Topic is empty. – Johnathon Sullinger Apr 04 '18 at 02:50
  • I updated my OP with a screenshot of the topic for reference. – Johnathon Sullinger Apr 04 '18 at 02:51
  • Yeah so the messages are available inside the subscription. Topic ingest the messages. you need to think at the subscription as a sub-queue of the topic. Have a look at this SO post to access deadletter queue https://stackoverflow.com/questions/22681954/how-do-you-access-the-dead-letter-sub-queue-on-an-azure-subscription – Thomas Apr 04 '18 at 02:52
  • Blah, so I'll have to write a completely separate tool to go fetch those messages and integrate it somehow into my runtime pipeline? It's really unfortunate that there isn't a way for me to force a re-delivery of those messages to the Azure Function. – Johnathon Sullinger Apr 04 '18 at 02:54
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/168160/discussion-between-thomas-and-johnathon-sullinger). – Thomas Apr 04 '18 at 02:54

1 Answers1

2

If the MaxDeliveryCount is reach the message will be sent to a deadletter queue.
You can think about a subscription as a sub-queue so you can still access these messages.

Check this post on how to access deadletter queues:

First thing is your scenario will be to increase the MaxDeliveryCount.

Then you can create a function that will be triggered every time a message arrived in the deadletter queue/subscription:

 [FunctionName("my-subscription-failure")]
 public static async Task RunFailure(
        [ServiceBusTrigger("%MytopicName%", "%MySubscription%/$DeadLetterQueue", Connection = "MyconnectionString")] BrokeredMessage message,
        [ServiceBus( "%MytopicName%", AccessRights.Send, Connection = "MyconnectionString")] ICollector<BrokeredMessage> queueBinding,
        TraceWriter log, ExecutionContext context)
{
    // fist check if the message can be resent (it could be a message that can't be replay)
    ...

    // If ok, resent the message
    queueBinding.Add(message);
}
Thomas
  • 24,234
  • 6
  • 81
  • 125