2

I am trying to create a Bigquery table as part of the dataflow. The examples show passing the schema as TableFieldSchema instance. However, the tableschema I have is data dependent, and hence can at best be created as an element in PCollection<TableFieldSchema>. For example:

PCollection<TableRow> quotes = ...;

  quotes.apply(BigQueryIO.Write
      .named("Write")
      .to("my-project:output.output_table")
      .withSchema(schema)
      .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE)
   .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));

Here schema needs to be a TableFieldSchema, but I have it as PCollection<TableFieldSchema>.

Mosha Pasumansky
  • 13,206
  • 5
  • 32
  • 55
user2254391
  • 330
  • 2
  • 11
  • Possible duplicate of [Writing different values to different BigQuery tables in Apache Beam](http://stackoverflow.com/questions/43505534/writing-different-values-to-different-bigquery-tables-in-apache-beam) – jkff May 04 '17 at 21:17

1 Answers1

3

We, unfortunately, don't have a built-in API to write to a BigQuery table with a dynamic schema. That said, we are working on improving flexibility in this area. No estimates at this time, but we hope to get this soon.

In the meanwhile, some workarounds have been proposed on other StackOverflow questions:

Community
  • 1
  • 1
Davor Bonaci
  • 1,709
  • 8
  • 9
  • Sorry to be a noob here. So, I can write a Custom Source, but I would still get a PCollection<TableFieldSchema>, rather than a tablefieldSchema. I guess I am struggling how to write a function something like this: `public static TableFieldSchema convFn(PCollection) schColl)` – user2254391 Aug 05 '15 at 06:43
  • I guess I am looking for something similar to Apache Spark's `rdd.collect()`. – user2254391 Aug 05 '15 at 08:24
  • Custom sources/sinks don't support side inputs yet, so it would likely be hard to pass PCollection and PCollection simultaneously. That said, if the schema can be computed within the custom sink, as opposed to being passed to it, that might be viable. – Davor Bonaci Aug 05 '15 at 21:14