4

I am getting the BigQuery table name at runtime and I pass that name to the BigQueryIO.write operation at the end of my pipeline to write to that table.

The code that I've written for it is:

rows.apply("write to BigQuery", BigQueryIO
    .writeTableRows()
    .withSchema(schema)
    .to("projectID:DatasetID."+tablename)
    .withWriteDisposition(WriteDisposition.WRITE_TRUNCATE)
    .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED));

With this syntax I always get an error, 

Exception in thread "main" java.lang.IllegalArgumentException: Table reference is not in [project_id]:[dataset_id].[table_id] format

How to pass the table name with the correct format when I don't know before hand which table it should put the data in? Any suggestions?

Thank You

rish0097
  • 1,024
  • 2
  • 18
  • 39
  • How many table can be there to write the data into bigquery? – Manoj Kumar Jul 11 '17 at 16:43
  • Where is the "tableName" field or variable coming from? How is it defined? – Ben Chambers Jul 11 '17 at 16:52
  • Also: this error message seems incomplete; I believe it should include the actual value of the table spec. It is difficult to help you with this error without knowing the exact value being passed to .to() - the error message simply says that the value you passed to .to() is malformed; it is not related to the fact that you are passing it at runtime. – jkff Jul 11 '17 at 17:57
  • Perhaps another thing you'll find useful: see this answer https://stackoverflow.com/a/43505535/278042 if the tables you're writing to are determined by the data itself. – jkff Jul 11 '17 at 17:57
  • @BenChambers What I'm doing is, I'm getting the "tableName" from a field in a BigQuery table and storing it in a String variable...which I'm ultimately passing to the BigQueryIO.write() operation. – rish0097 Jul 12 '17 at 07:26
  • What value does `tablename` have at the point in the code you've shown? – Ben Chambers Jul 12 '17 at 16:07

1 Answers1

3

Very late to the party on this however. I suspect the issue is you were passing in a string not a table reference.

If you created a table reference I suspect you'd have no issues with the above code.

com.google.api.services.bigquery.model.TableReference table = new TableReference()
            .setProjectId(projectID)
            .setDatasetId(DatasetID)
            .setTableId(tablename);

rows.apply("write to BigQuery", BigQueryIO
    .writeTableRows()
    .withSchema(schema)
    .to(table)
    .withWriteDisposition(WriteDisposition.WRITE_TRUNCATE)
    .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED));
Campey
  • 1,108
  • 9
  • 15
  • Hi Campey...yeah but the issue here is that i think you won't be able to pass the valueProvider within .setTableId(tablename). Currently, BigQueryIO directly allows to pass ValueProvider while writng to BigQuery. This is useful if we have the entire location. Otherwise there are options like Dynamic Destinations or the one that @jkff has recommended through the link he provided https://stackoverflow.com/a/43505535/278042. – rish0097 May 28 '18 at 06:59
  • I had done it through Dynamic Destinations. Will provide the solution in just a bit. – rish0097 May 28 '18 at 07:00