0

I want to use the BigQueryDataTransferServiceStartTransferRunsOperator based on this example: https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery_dts.html

I want to pass the path of the CSV file (or folder with templated path) from Airflow to this operator somehow and execute the existing BigQuery Transfer. The problem is, that I can't find any solution to pass Runtime Arguments to this operator.

I don't want to recreate this BQ transfer every time with BigQueryCreateDataTransferOperator and add a new config, because this way I'll loose the Transfer Service logs from previuos days.

What I want is to somehow define the Transfer Service with a template string for the input path and fill this template when the DAG runs, or pass the path of the CSV file directly.

Is it possible to somehow achieve this? If yes, how?

elaspog
  • 1,635
  • 3
  • 21
  • 51
  • Have you try this approach? you can create a `pythonoperator` before your `bigquerydatatransferoperator` which the output of that operator will be the csv path file... then you can use the output parameter for the next operator and run it. – Betjens Nov 24 '21 at 12:59
  • Here are the links: [TaskFlowAPI](https://stackoverflow.com/questions/69986891/how-to-play-a-base64-response-from-google-text-to-speech-as-an-mp3-audio-in-brow) and [Operators](https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#operators) – Betjens Nov 24 '21 at 13:02

0 Answers0