8

I have a Google Cloud Pub/Sub subscription that is using a dead-letter topic. I recently had an outage that was preventing a number of messages from being processed & they ended up in the dead-letter topic.

The outage was resolved and I'd like to easily send the contents of dead letter subscription back to the original subscription. They're all present in the queue still (I have nothing consuming the dead letter sub) so I just need to route them somewhere.

This is an admin task so I'd like it to be manually initiated, if that makes any difference. Ideally via the UI but I can't see anything there.

Mark McDonald
  • 7,571
  • 6
  • 46
  • 53
  • How are you processing your messages ? You can manually start a new batch job consuming message from dead letter topic. – kaysush May 07 '20 at 05:05
  • have you created a pull subscription on your deadletter topic? In addition, is it a problem is you re-process already processed messages? – guillaume blaquiere May 07 '20 at 07:56
  • Was your problem solved? – rmesteves May 08 '20 at 16:04
  • No pull subscription has been configured, the messages stay in the queue for manual inspection without ack. – Mark McDonald May 28 '20 at 08:50
  • Keep in mind that if you deliver back to the topic, all subscribers will get the message again which, if you're consumers are not designed to be idempotent, could be problematic. I wish there were a way to redeliver to the original subscription which is a request I put in with the product team a couple times now. I think in the meanwhile, the "best practice" is actually to process the dead-letter subscription before switching to the main subscription on startup. – Sinaesthetic Mar 16 '23 at 16:01

2 Answers2

5

You have a few options:

  • Use a Dataflow pipeline to move messages from dead letter topic to your topic.
  • Update existing pipeline to read from both original topic and dead letter topic based on configuration
  • Create a new system that, when enabled, moves messages from one topic to another.

The right answer might depend on your system design and requirements.

In case your use case for dead letter topic always includes moving the messages back to the primary topic after a delay, you might want to use configurable exponential backoff in Cloud Pub/Sub. This feature will have general availability towards the end of Q2 2020.

Mahesh Gattani
  • 321
  • 1
  • 4
  • Interesting - I like the RetryPolicy, would that be configurable through the UI once GA? In this case, the messages were undelivered because of an issue that lasted ~hours, so they were shunted off to the DLQ. I think your first option is the best, if I can define and run them ad-hoc. Thanks for the direct link. – Mark McDonald May 28 '20 at 08:55
  • RetryPolicy will be configurable through the UI once GA. – Mahesh Gattani Jun 02 '20 at 15:29
  • 1
    You can now configure a retry policy in your Cloud Pub/Sub subscription. Retry policy allows users to have per message exponential backoff in case of failures. More information can be found [here](https://cloud.google.com/pubsub/docs/admin). – Mahesh Gattani Jun 16 '20 at 15:07
  • Sounds to me like one could implement your "moving the messages back to the primary topic after a delay" in a way that requires no manual intervention by setting the dead letter topic of the dead letter subscription to the primary topic. An application whose only purpose would be to nack messages from the dead letter subscription would be running always. It would be an infinite loop, but a long delay in the retry policy of the dead letter subscription would mean after a long period of time, the messages would eventually end up back in the primary subscription to be reprocessed. – Matt Welke Jun 20 '21 at 19:57
  • hmmm I have to say that this seems pretty friggin' nuts that the base case use of a DLQ must be handled in such a convoluted way... I'm wondering if someone has made a CLI script to accomplish this, I've seen it for AWS's SQS in their docs – SirRodge May 21 '23 at 05:16
  • Actually here looks like a good reference to craft a script from: https://github.com/samswen/gcp-utils/blob/master/src/index.js – SirRodge May 21 '23 at 05:23
  • okay - I went ahead and made a script here: https://stackoverflow.com/questions/61650087/how-to-route-dead-letter-messages-back-to-the-original-topic/76301420#answer-76301420 – SirRodge May 21 '23 at 18:50
1

For the frustrated few who may come across this issue, I went ahead and made a Node script here for republishing messages from subscription back to topic that you can hopefully easily adapt for your own use.

It can of course be used this DLQ --> main republish scenario but also any other scenario where you may want to manually republish from subscriber to topic.

https://github.com/sirrodgepodge/gcp-pubsub-republish/blob/main/index.ts

Note that the Dataflow pipeline approach mentioned in Mahesh Gattani's answer does work but was going to cost $120/month per queue and that felt like a bit much for such a basic need.

SirRodge
  • 594
  • 5
  • 8