3

My customer would like to use a CosmosDBTrigger to transfer documents to an Azure Service Bus. In this scenario it's important to have a 1:1 relationship between Cosmos item mutations and Service Bus messages. So, each document that the trigger receives (via batch) must be processed only once, this leads to some fundamental questions I've been unable to confirm:

  • What happens if the code throws an exception during processing? Will the batch simply be forgotten?
  • Similarly if there is a runtime issue with the function service, or the CosmosDBTrigger library that causes an interruption prior to calling the Azure Function, can this be recovered without skipping the batch or duplicating documents?
  • Is there any way for the function to report the completion status, such as "not processed"; causing the trigger to retry the batch?
  • Are there circumstances where the same document will be processed multiple times, either by the same or a different instance? (I've read a couple of posts claiming this occurred.)

If the net answer is that this trigger is not reliable, just curious what are the intended use cases for its use?

Thanks

-John

JohnKoz
  • 908
  • 11
  • 21

1 Answers1

3

Short answer is no. The Cosmos DB Trigger has an "at least once" delivery, which means that an item could be, in some cases, delivered more than once.

  1. As per the link below (and aligning with other Event based Azure Functions Triggers), unhandled exceptions won't cause the batch to retry. There is a design proposal from the Functions team to make all Event based Triggers have a retry configuration (https://github.com/jeffhollan/retry-design), once that is applied, then you will be able to define a retry policy for the Cosmos DB Trigger too.
  2. If there is a runtime issue that stops the current batch at any point in your Function code, when the runtime starts again, it will retry the batch entire batch, there is no way to know up to which point in the batch you read or processed, because the Function user code is private. The life cycle of the Function checkpoints on the lease store after a complete execution of the Function, if the runtime stops in the middle (because you manually stopped it or due to some event that stopped the runtime), then the checkpoint did not happen, and the lease store has the previous mark.
  3. As mentioned previously, there is no retry behavior on Event-based triggers, so there is no way you can retry it based on some condition.
  4. Yes, as mentioned, the Trigger has an "at least once" delivery. The same document could be processed again if the amount of instances is changing due to an scaling event, where the Trigger will rebalance the leases across new instances.

For reference, see the Troubleshooting guide: https://learn.microsoft.com/azure/cosmos-db/troubleshoot-changefeed-functions.

Matias Quaranta
  • 13,907
  • 1
  • 22
  • 47
  • 1
    Thank you Matias. To clarify #1 - if the function throws an exception (unhandled or deliberate) the trigger will absorb the error and complete the batch, moving on to the next changes. And if the runtime fails before completing the batch, the entire batch will be replayed when the function recovers. – JohnKoz May 12 '20 at 16:16
  • 1
    Correct. The key here is if the Runtime gets to the point when the checkpoint is saved. If the Runtime is running, then it will checkpoint, regardless of the user Function result. If the Runtime goes down, that points to a bigger issue and the code just won't get to checkpoint. – Matias Quaranta May 12 '20 at 16:41
  • 1
    Claiming "at least once" is false and dangerous. If your destination is down, you can't do anything in your function but do a `FailFast()` and crash the process. If you throw an exception you have data loss as the Trigger will continue with the next batch of changes. If you need reliable processing do not use a CosmosDBTrigger!!!. Instead implement and consume the Cosmos Change feed from a Web App or Continous WebJob by using the existing ChangeFeed API which do implement "at least once" delivery. – CodeMonkey Jul 14 '21 at 10:45