GCP Data Fusion multiple table import

Question

I'm trying to use Multiple Database Tables and BigQuery Multi Table Data Fusion plugin to import multiple table in one pipeline

But when I try to execute I get the following error

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BigQuery Multi Table has no outputs. Please check that the sink calls addOutput at some point.

I'm using Data Fusion version 6.1.4 Multiple Database Tables version 1.2.0 and BigQuery Multi Table version 0.14.8.

Any suggestion on what may be the problem?

Edit:

following the configuration of multiple table database source

{
            "name": "Multiple Database Tables",
            "plugin": {
                "name": "MultiTableDatabase",
                "type": "batchsource",
                "label": "Multiple Database Tables",
                "artifact": {
                    "name": "multi-table-plugins",
                    "version": "1.2.0",
                    "scope": "USER"
                },
                "properties": {
                    "splitsPerTable": "1",
                    "referenceName": "multiTable",
                    "connectionString": "${secure(connection)}",
                    "jdbcPluginName": "netezza",
                    "user": "${secure(username)}",
                    "password": "${secure(password)}",
                    "whiteList": "categoria_l,cliente_l,regione_l"
                }
            },
            "outputSchema": [
                {
                    "name": "etlSchemaBody",
                    "schema": ""
                }
            ]
        },

After further test the problem is that the source response is empty because data fusion is not reading view from source database but only tables

Did your pipeline finish successfully, populating the tables to Bigquery sink? — Nick_Kh, Oct 15 '20 at 11:14
It seems that `BigQuery Multi Table` plugin doesn't produce any Output schema but basically spreads Bigquery Job to insert/update columns across the tables. — Nick_Kh, Oct 15 '20 at 11:24
The data previously is stored in temporary GCS bucket, and then imported to Bigquery tables. — Nick_Kh, Oct 15 '20 at 11:34
@Nick_Kh no if i run the pipeline fails with the error reported in the question — Paolo Fusari, Oct 15 '20 at 12:33

xgMz · Accepted Answer · 2020-10-16T16:31:21.050

2

It seems like the Multiple Database Tables source produced no records ("Out 0"). I'd check there first. You can do a quick check using the Preview mode. Plugin doc here.

Related answer here.

edited Oct 16 '20 at 16:31

answered Oct 16 '20 at 16:11

xgMz

3,334
2
30
23

The source tables are not empty because I have a pipeline for each table that imports data correctly. Since each table has few rows I tried to use multiple table to reduce import time. – Paolo Fusari Oct 19 '20 at 08:30
I added the configuration of Multiple Database Tables source. I think the problem may be that I haven't set Table Name Pattern but where i can find the syntax for pattern? – Paolo Fusari Oct 19 '20 at 13:13
After further testing the problem was the plug in empty response of multiple table source because seems reading only tables and not views. – Paolo Fusari Oct 20 '20 at 07:34
1

I changed the plugin to also allow Views - here's the diff: https://github.com/data-integrations/multi-table-plugins/compare/develop...carlosmarin:source-views and here's the artifact: https://github.com/carlosmarin/multi-table-views-plugins-1.3.5 To use it you'll have to upload the JAR using the big green plus sign in Data Studio. – xgMz Oct 20 '20 at 15:04

GCP Data Fusion multiple table import

1 Answers1

Linked