3

I was building a service that runs on Cloud Run that is triggered by PubSub through EventArc. 'PubSub' guarantees delivery at least one time and it would retry for every acknowledgement deadline. This deadline is set in the queue subscription details.

We could send an acknowledgement back at two points when a service receives a pub-sub request (which is received as a POST request in the service).

  1. At the beginning of the request as soon as the request was received. The service would then continue to process the request at its own pace. However, this article points out that

When an application running on Cloud Run finishes handling a request, the container instance's access to CPU will be disabled or severely limited. Therefore, you should not start background threads or routines that run outside the scope of the request handlers.

So sending a response at the beginning may not be an option

  1. After the request has been processed by the service. So this would mean that, depending on what the service would do, we cannot always predict how long it would take to process the request. Hence we cannot set the Acknowledgement deadline correctly, resulting in PubSub retries and duplicate requests.

So what is the best practice here? Is there a better way to handle this?

ahmet alp balkan
  • 42,679
  • 38
  • 138
  • 214
vzurd
  • 1,416
  • 2
  • 15
  • 37

3 Answers3

2

Best practice is generally to ack a message once the processing is complete. In addition to the Cloud Run limitation you linked, consider that if the endpoint acked a message immediately upon receipt and then an error occurred in processing it, your application could lose that message.

To minimize duplicates, you can set the ack deadline to an upper bound of the processing time. (If your endpoint ends up processing messages faster than this, the ack deadline won’t rate-limit incoming messages.) If the 600s deadline is not sufficient, you could consider writing the message to some persistent storage and then acking it. Then, a separate worker can asynchronously process the messages from persistent storage.

Allison Fisher
  • 143
  • 4
1

Since you are concerned that you might not be able to set the correct "Acknowledgement Deadline", you can use modify_ack_deadline() in your code where you can dynamically extend your deadline if the process is still running. You can refer to this document for sample code implementations.

Be wary that the maximum acknowledgement deadline is 600 seconds. Just make sure that your processing in cloud run does not exceed the said limit.

Ricco D
  • 6,873
  • 1
  • 8
  • 18
0

Acknowledgements do not apply to Cloud Run, because acks are for "pull subscriptions" where a process is continuously pulling the Cloud PubSub API.

To get events from PubSub into Cloud Run, you use "push subscriptions" where PubSub makes an HTTP request to Cloud Run, and waits for it to finish.

In this push scenario, PubSub already knows it made you a request (you received the event) so it does not need an acknowledgement about the receipt of the message. However, if your request sends a faulty response code (e.g. http 500) PubSub will make another request to retry (and this is configurable on the Push Subscription itself).

ahmet alp balkan
  • 42,679
  • 38
  • 138
  • 214
  • "Acknowledgements do not apply to Cloud Run" - not true. I can see pubsub retry a message to cloud run when the deadline is breached. – vzurd Jun 26 '21 at 08:07
  • Acks apply to push and pull subscriptions, but you can’t use ModifyAckDeadline for push subscriptions (and push delivery rate is auto-flow-controlled instead; see https://cloud.google.com/pubsub/docs/push#quotas_limits_and_delivery_rate) – Allison Fisher Jun 28 '21 at 17:28
  • In this case, the "ack" is not to signal "I am processing this message", but rather to signal its completion through HTTP response. If `ackDeadlineSeconds` is 10 minutes, another subscriber will get the message if you can't return HTTP 200 OK by then. If you return a failure earlier, another subscriber may get it right away. That's what I meant here. I don't think there's anything to downvote... – ahmet alp balkan Jun 28 '21 at 21:11