0

We are looking to build a solution on GCP for campaign/ad analytics (ingest doubleclik and other ad-server data into DW). Data is ingested as batch, with star schema but will have updates trickling in for up to a week, need trend analysis for multiple clients (advertisers) and reporting. I can't decide between Google Big table which supports updates and timeseries analysis Vs Big Query which is ideal for star schema and ad-hoc analysis.

Any suggestions? Performance and flexibility are important.

Mosha Pasumansky
  • 13,206
  • 5
  • 32
  • 55
Tim S
  • 185
  • 1
  • 13
  • 1
    This past thread may help: http://stackoverflow.com/questions/34437572/google-bigtable-vs-bigquery-for-storing-large-number-of-events. Note that BigQuery does allow for updates too; in addition to appending data, you can use DML to modify existing data if you need: https://code.google.com/p/google-bigquery/issues/detail?id=657. – Elliott Brossard Sep 26 '16 at 06:23
  • I believe they have a cloud connector (might still be in beta) that can automatically dump DFP logs into BigQuery in a similar manner to how they dump GA data into daily tables for GA premium subscribers. I'd reach out to someone from Google cloud if you can. – andrewm4894 Sep 26 '16 at 13:35
  • Also there are some managed service approaches to this with people like switchboard-software.com I've dealt with them and they know this stuff inside out. – andrewm4894 Sep 26 '16 at 13:37
  • P.s I'd go with Google data flow (apache beam) + big query – andrewm4894 Sep 26 '16 at 13:38

1 Answers1

1

You may find the following solution guide helpful for learning how to build an analysis pipeline using BigQuery and other GCP products and tools:

Bigtable meanwhile is a good fit for building real-time bidding and other pieces of core ad serving infrastructure. See e.g.:

Ramesh Dharan
  • 895
  • 4
  • 14