0

We are Trying to copy data from Google Cloud DataStore to BigQuery by using Compute Engine VM Instance on daily basis,but its so costly to me copy whole data set to BigQuery, Basically we Required updated data only (the record which has changed only) we don't want to copy whole table from datastore to bigquery by using shell script. please help us to resolve this issue...

Abhinav Bihade
  • 3
  • 3
  • Why don't you try only writing the updates to DataStore, then bring them into BigQuery with a merge command. https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#merge_statement – Ben P Feb 10 '20 at 13:30

1 Answers1

0

when you export data from datastore to Bigquery you cannot append data to an existing table. you can either create a new table or overwrite an existing table. Either way you have to export all your entities or entities of specific kind from your datastore but you cannot export just the new data.

an example script that can handle export data from datastore to Bigquery can be found here.

If you want to reduce cost use: - preemtibale instances which is very cheap in comparison to normal instances --> for cron jobs

Another way that I found is this. but I'm not sure if it would work because it's an old post and it uses MapReduce API.

Methkal Khalawi
  • 2,368
  • 1
  • 8
  • 13