0

We are planning to use GCP Pubsub to write events to GCS. I have the below questions.

  1. We want to enable the audit table in BigQuery, we would like to see how many messages came for the particular time frame. By day, hour

  2. How do we validate from Pubsub let's say we received 10 messages, how do we check against GCS? How to check we didn't drop any messages.

I would really appreciate your feedback.

user138770
  • 15
  • 5

1 Answers1

0

To validate number of records written written to GCS, you can create Big query external temp table and query for number of records written to GCS. This sanity check need to be done at regular interval.

Second solution :- You can also check no or records written to GCS through following command:-

gsutil cat gs://folder/test.csv | wc -l
Vibhor Gupta
  • 670
  • 7
  • 16
  • Thank you for the suggestion. Okay, I can check the number of records from GCS but how to check the number of records in the particular message from Pubsub to tie back with GCS? Is there a way to know the number of records in one message so that I can validate against GCS? – user138770 Jan 19 '21 at 02:39
  • May be you can have a look to following case (https://stackoverflow.com/questions/35475082/google-pubsub-counting-messages-in-topic), it may help you. – Vibhor Gupta Jan 19 '21 at 08:22