I have set up a service and is working as intended however every time the service runs, there are new records that show up in the table. I want to edit the configuration to allow the service to replace all records in the destination table instead of appending records.
I don't want to use partitioned tables as I only want to use the latest data available.
But the documentation doesn't have any option to replace records instead of appending records: https://cloud.google.com/bigquery-transfer/docs/s3-transfer-intro
They do mention: Transfers from Amazon S3 are always triggered with the WRITE_APPEND preference which appends data to the destination table. See configuration.load.writeDisposition in the load job configuration for additional details.
https://cloud.google.com/bigquery-transfer/docs/s3-transfer
I don't provide see any details about "configuration.load.writeDisposition". Should I be learning the command line in order to do this?
