0

I'm new to building pipelines, and I was using the twitter api to stream data. My current approach is as follows:

  1. I send the streamed data to a Topic on pub/sub, which triggers the cloud function to insert the message into BigQuery after processing.
  2. The other approach recommended was using pub/sub + DataFlow, but I'm not sure how that makes a difference.

Can someone elaborate on the pros and cons of the two approaches ?

Mikhail Berlyant
  • 165,386
  • 8
  • 154
  • 230
Atharva Chalke
  • 1
  • 1
  • 2
    Does this answer your question? [Benefits with Dataflow over cloud functions when moving data?](https://stackoverflow.com/questions/51197653/benefits-with-dataflow-over-cloud-functions-when-moving-data) – Nick_Kh Mar 27 '20 at 09:12
  • That is perfect. Thank you ! – Atharva Chalke Mar 27 '20 at 09:16

0 Answers0