0

I am using GCP to process and store data sent from a simulated IoT device. The data (should) flow in the following path:

[Simulated IoT device with mqtt client] -> [IoT core device] -> [IoT core registry] -> [pub/sub topic] -> [pub/sub subscription] -> [cloud dataflow pub/sub subscription to BigQuery Template] -> [BigQuery Table]

The company I am has some configuration of GCP (it is on a VPC), which requires that I specify a subnet for DataFlow to even run . When dataflow runs after specifying a subnet it doesn't do anything. It doesn't pull data from the subscription and does not output any error messages. It is apparently unaware that data is being sent to the topic and subscription. I have also tried a cloud dataflow pub/sub topic to BigQuery Template and that also does nothing. I have verified that data is being sent to the subscription.

What is the best way to to troubleshoot this issue? I'm at a loss on why there is no communication between these two components.

dda2120
  • 41
  • 1
  • 5
  • I think we want to delve into the nature of the subscription. Is it a pull subscription where the client effectively pulls the messages or is a push subscription where GCP sends the messages as REST requests? Are the messages accumulating against the subscription? If not, this may indicate that "something" is retrieving them but not what you think. If I were in your shoes, I'd suspend any consumers of the subscription and check that messages are accumulating. I'd then test with gcloud that they are available for retrieval as desired. – Kolban Oct 01 '21 at 23:37
  • It is a pull subscription. The messages accumulate and are not retrieved. I have also tried the Dataflow pub/sub topic to BigQuery template and I also observe messages accumulating in the subscription created by that template. I have also been able to manually retrieve the messages using the GCP CLI. – dda2120 Oct 02 '21 at 00:24

1 Answers1

0

I was able to finally figure out the issue. The problem was that the region for the virtual private cloud did not match the region for dataflow. I also was not able to use the cloud dataflow pub/sub subscription to BigQuery Template, but I was able to use the cloud dataflow pub/sub topic to BigQuery.

The main takeaway is that the regions matter in this context and if they do not match nothing will happen. If nothing is happening, double check that you have selected the correct regions.

dda2120
  • 41
  • 1
  • 5