0

I'm trying to ingest data from different tables with in same database using Data fusion Multiple database tables plugin to bigquery tables using multiple big query tables sink. I write 3 different custom SQL and add them inside the plugin section which is under "Data Section Mode" > "Custom SQL Statements".

The problem is When I preview or deploy and run the pipeline I get the error "BigQuery Multi Table has no outputs. Please check that the sink calls addOutput at some point."

What I try to figure out this problem;

  • Run custom SQL on database and worked properly.
  • Create pipelines that are specific for custom SQLs but it's like 1 table ingestion from sql server to bigquery table as sink. it worked properly.
  • Try different Data Section Mode under multiple database tables plugin that is Table Allow List , works but it's just insert all data with no option to transform any column or filtering. Did that one to see if plugin can reach the database and able to read data ,it can read. Data Pipeline - Multiple Database Tables Plugin Config - 1 Data Pipeline - Multiple Database Tables Plugin Config - 2

As a conclusion I would like to ingest data from one database with multiple tables with in one data pipeline. If possible I would like to do it with writing custom sqls for each tables.

Open for any advice and try. Thank you.

wweer
  • 5
  • 3
  • Similar question - https://stackoverflow.com/questions/64368503/gcp-data-fusion-multiple-table-import – dikesh Jul 20 '22 at 09:20
  • Thank you but I try to import tables not views – wweer Jul 20 '22 at 09:28
  • Try enabling the 'Allow flexible schemas in Output' property in the sink. This should allow the sink to detect the different table schemas dynamically at runtime instead of trying to look them up at the start of the run. – Albert Shau Sep 06 '22 at 21:03

0 Answers0