0

I have a dead-letter queue for a pubsub cloud function that is receiving messages using PUSH subscription and at the moment the service is successfully sending those messages to the dead-letter topic and the dead-letter subscription when fails. However, I am uncertain on how to carry on once it reaches this dead-letter subscription.

I don't want to lose the messages that have been sent to the dead letter so my idea would be that in case of failures from the service to acknowledge the message after predefined delivery attempts, the message will be forwarded to a dead letter topic. The same service, when back to life, can pull the messages from the dead-letter topic as well to see what it missed during the times of unavailability.

There is a similar post in here but the answer only points to the options but not the solutions, and unfortunately, I haven't been able to find it.

There is also a mention in here about this issue where it's actually from where I have taken my question.

Please, could somebody point me in the right direction? Is there a better way?

AudronFS
  • 139
  • 7

1 Answers1

1

If the main process aren't able to process the messages, you have to rely on the retry mechanism of PubSub.

If you put the messages in a Dead letter topic, it's because you can't process the message with the main function. So, it's another process, another function. You can't process with the same function the dead letter message (In fact you can, but it's an anti pattern).

A good pattern is to save your dead letter message somewhere. When your main function is up again, trigger a process that read the messages and re publish the messages in the main topic.

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • Thank you Guillaume, sometimes you have temporary issues with the function i.e. function is calling an API that is returning 500. I guess and as you point out, you have to rely on something like the exponential back-off. However, in my experience, this is not always enough, that's why and taking into account that it's an issue already mentioned by other sources, I asked this question. – AudronFS Nov 26 '21 at 08:49
  • In the past, I used DataStore to be sure these messages are not lost, just thought this could potentially simplify the process. It could be a great feature to have some sort of configuration in which you can retain the messages in the topic until some condition is accomplished. – AudronFS Nov 26 '21 at 08:49