Stack Overflow
Stack Overflow

Questions tagged [google-cloud-data-transfer]

Use this tag for Google Cloud Platform Data Transfer products (https://cloud.google.com/products/data-transfer/) which help customers move unstructured data between Google Cloud and other clouds or storage systems in private data centers. This includes products like Storage Transfer Service, Transfer Service for on premises data, Transfer Appliance and the use of the gsutil cp command.

71 questions
4
votes
0 answers

Transferring files modified in the last 10 minutes from S3 to Big Query using Big Query Data Transfer Service

On Big Query's Data Transfer Service Documentation for S3, it states: To reduce the possibility of missing data, schedule your Amazon S3 transfers to occur at least 10 minutes after your files are added to the bucket. By default, when running a…
Jianying Chiang
  • 81
  • 4
4
votes
3 answers

Automatically retrieving large files via public HTTP into Google Cloud Storage

For weather processing purpose, I am looking to retrieve automatically daily weather forecast data in Google Cloud Storage. The files are available on public HTTP URL (http://dcpc-nwp.meteo.fr/openwis-user-portal/srv/en/main.home), but they are very…
Matthieu
  • 71
  • 5
3
votes
3 answers

Cloud Storage Transfer Service from S3 stuck in "this transfer is starting"

A part of our application initiates transfers from Amazon S3 to Google Cloud Storage via the storage transfer service API. We've had this up and running successfully for several months until yesterday when our transfers stopped working. We can see a…
Chris Alme
  • 81
  • 6
2
votes
0 answers

YouTube Content Owner video metadata report - no reports for reporting job

I'm using a BigQuery Data Transfer to load YouTube Content Owner data into BigQuery. The transfers run daily. Even though transfers have been successful, there is no new data arriving for content owner video metadata reports…
m.nicolau
  • 47
  • 5
2
votes
1 answer

DataTransferService API - Module 'google.cloud.bigquery_datatransfer_v1.types' has no 'Timestamp'

I want to test BigQuery DataTransfer API in local but I encounter problems with library : client = bigquery_datatransfer_v1.DataTransferServiceClient() projectid = 'MyProjectID' transferid = 'MyTransferID' parent =…
Sara
  • 21
  • 2
2
votes
0 answers

BigQuery Data Transfer Service with BigQuery partitioned table

I have access to a project within BigQuery. I'm looking to create a partitioned table by ingestion time, partitioned by day, then set up a BigQuery Data Transfers process that brings avro files in from multiple directories within a Google Cloud…
MRHarv
  • 491
  • 1
  • 10
  • 22
2
votes
0 answers

Why is google ads transfer data changing outside of 30 day attribution window?

I have set up a google ads transfer into bigquery with a refresh window set at 30 days. I am aware that conversion-type statistics are continually updated within the lookback window, so I am not concerned about discord between the data pull and the…
Riley Runnoe
  • 68
  • 5
2
votes
1 answer

Move files from Regional to Multiregional Bucket in GCE

I have a "nearline" + "europe-west4 (Netherlands)" bucket in the Google Computer Engine. If the files in the bucket reach a certain age (say, 6 months), I'd like to move them to a "coldline" + "mutli-regional (EU)" bucket. I tried implementing this…
r0f1
  • 2,717
  • 3
  • 26
  • 39
1
vote
1 answer

Get details of All Class A operations in GCP Cloud Storage to Understand Cost

I am running a large data transfer between buckets using data transfer service for cloud storage. I want to find out exactly how many times I am invoking ClassA operations and ClassB operations to understand and analyse cost trends. Would like to…
SRJ
  • 2,092
  • 3
  • 17
  • 36
1
vote
1 answer

No Cloud Storage Audit Logs for Data Transfer Service

I used data transfer service for transferring large amount of data from one bucket to another. I have enabled cloud storage auditing for the project. I have enabled data access audit logs for cloud storage. Admin Read Data Read Data…
SRJ
  • 2,092
  • 3
  • 17
  • 36
1
vote
0 answers

Alternative for skip operation's cost overhead : --no-clobber in CP operation of GCP

I am using gcloud storage cp for transferring large amount of data from source bucket to destination bucket. I am using --no-clobber option to skip the existing file if copied already. gcloud storage cp -r --no-clobber "gs://test-1/*" "gs://test-2"…
SRJ
  • 2,092
  • 3
  • 17
  • 36
1
vote
1 answer

How to get bigquery's scheduled queries execution log programmatically?

I would like to get execution log of scheduled queries in bigquery programmatically. It is very much possible using web UI but if there any way using Standard SQL or any API which is providing that log would be fine. Thanks in advance
vignesh
  • 1,414
  • 5
  • 19
  • 38
1
vote
1 answer

Parquet with Null Value for column is converted to Integer

I'm using python pandas to write a DataFrame to parquet in GCS, then using Bigquery Transfer Service to transfer the GCS parquet file to a Bigquery table. Sometimes when the DataFrame is small, an entire column might have NULL values. When this…
Josh Wang
  • 521
  • 7
  • 17
1
vote
0 answers

bigquery s3 data transfer service edit parameters to remove default appending records

I have set up a service and is working as intended however every time the service runs, there are new records that show up in the table. I want to edit the configuration to allow the service to replace all records in the destination table instead of…
Harpreet Gill
  • 11
  • 1
1
vote
1 answer

Google Ads transfer to BigQuery misses the data from Smart Campaigns

I have a Google Ads account which has a single Smart Campaign and multiple usual campaigns. Also I've set up a data transfer to Google BigQuery. When I try to compare BigQuery data using the query SELECT sum(Cost) FROM…
Nikolay Suschenko
  • 13
  • 2
1
2 3 4 5