40

For pubsub topics with number of messages in the range of ~100k, what is the best practice for draining/dropping/clearing/deleting all messages using gcloud-java SDK?

Possible solutions:

  • Deleting and recreating the subscribers and then the publishers

  • High concurrency pull+ack (easy to hit the quota this way)

  • Something else

My hope is that this process can be fast (not more than ~60 seconds, say), robust, and uses supported SDK methods with minimal other code.

DrMarshall
  • 1,034
  • 2
  • 9
  • 14

1 Answers1

69

Update with description of snapshot and seek feature: One can use seek on a Pub/Sub subscription to ack older messages by seeking to a timestamp corresponding to now. The best way is via the gcloud command line tool. The command to acknowledge messages published up to a particular timestamp would be:

gcloud pubsub subscriptions seek <subscription path> --time=yyyy-mm-ddThh:mm:ss

To delete all messages up to now:

gcloud pubsub subscriptions seek <subscription path> --time=$(date +%Y-%m-%dT%H:%M:%S)

Previous answer prior to the addition of snapshot and seek: Currently, Google Cloud Pub/Sub has no way clear older messages, though it is something we are looking to add. Deleting and recreating the subscription would be the most efficient way to clear it, both in terms of time and cost. You wouldn't have to do anything with your publishers; any messages published from the point after recreation on will be sent to subscribers on the recreated subscription.

Kamal Aboul-Hosn
  • 15,111
  • 1
  • 34
  • 46
  • Deleting and recreating the subscriptions doesn't seem to be clearing messages already published to pub/sub. Deleting the topic and re-creating it also doesn't do it. Seems like the only sure fire way is to abandon the topic and create a new one altogether? That would require changing all the publishers as well. – Plasty Grove Jan 11 '18 at 14:42
  • When you say "doesn't seem to be clearing messages," what do you mean? The new subscription does not receive these messages, correct? – Kamal Aboul-Hosn Jan 11 '18 at 17:06
  • My bad - there was a bug with my dataflow subscriber. Looks like if I delete the subscriptions and re-create them, then only new messages get sent. – Plasty Grove Jan 12 '18 at 01:24
  • 1
    @KamalAboul-Hosn, is adding this still on Google's radar? Or is there a better way today? – Johan Walles Jun 12 '18 at 05:48
  • 1
    This is really a terrible option: delete and recreate the pubsub topic, really ?! – Stephane Paquet Aug 10 '18 at 20:19
  • I have updated my answer with information about the new seek API, which provides a way to ack previously published messages without deleting and recreating the subscription. – Kamal Aboul-Hosn Aug 13 '18 at 19:47
  • Delete and recreate subscription will clear messages. Test it at 2018.10.15 – Lin Du Oct 15 '18 at 03:10
  • `seek` command is not existed any more. `Google Cloud SDK 202.0.0` – Lin Du Oct 15 '18 at 03:21
  • Nice: Deleting the subs really wasn't a good method – Illegal Operator Nov 20 '18 at 13:17
  • `seek` command exists in 222.0.0: `gcloud alpha pubsub subscriptions seek --help`. You can also use the "Seek" button under subscription details in the pubsub google page. – Oliver I Nov 28 '18 at 19:19
  • 10
    Perhaps it's useful for some people for copy/pasting in terminal: `gcloud beta pubsub subscriptions seek testSubscription --time=$(date +%Y-%m-%dT%H:%M:%S)` – lumaks Jun 14 '19 at 21:15
  • Here's a Powershell version if anyone wants it `gcloud pubsub subscriptions seek testSubscription --time=$(Get-Date -format "yyyy-MM-ddThh:mm:ss")` – Michael Galos Mar 25 '21 at 21:53
  • The GCP UI has the button to purge a subscription, wondering when it will come to the clients? As using seek looks more like a workaround – Tudor Plugaru May 25 '21 at 08:32
  • If you need to drain messages older than N seconds, use `gcloud pubsub subscriptions update --message-retention-duration=m`, wait a few minutes for them to drop, then set your message retention duration back to the default value (1 week). – Joseph Lust Oct 21 '22 at 00:17
  • Changing the retention period and then changing jit back would not be considered the recommended way to drop a backlog and is not guaranteed to prevent the delivery of older messages. – Kamal Aboul-Hosn Oct 21 '22 at 17:43