1

I am following the instructions in the following post to write to a date-partitioned table in BigQuery. I am using a serializable function to map the the window to a partition-location using the $ syntax and I get the following error:

Invalid table ID \"table$19700822\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long.

Am I missing something here?

Edit adding code:

p.apply(Window.<TableRow>into(FixedWindows.of(Duration.standardDays(1))))
    .apply(BigQueryIO.Write
    .named("Write")
    .withSchema(schema)
    .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
    .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
    .to(new SerializableFunction<BoundedWindow, String>() {
      public String apply(BoundedWindow window) {
        String dayString = DateTimeFormat.forPattern("yyyyMMdd")
             .withZone(DateTimeZone.UTC)
             .print(((IntervalWindow) window).start());
        return "project_id:dataset.table$" + dayString;
      }
    }));
Community
  • 1
  • 1
Narek
  • 548
  • 6
  • 26

1 Answers1

2

Make sure that the table you're trying to access already exists. You can't create a table with "$" in it, and you're using "create if needed", so that your code might end up creating the table in addition to trying to write to it.

Sonya
  • 895
  • 6
  • 9
  • Looks like a combination of this an using `DataflowPipelineRunner` fixed the issue. – Narek Dec 28 '16 at 03:09
  • @Narek did you manage to create a BQ table with date partition from dataflow pipeline? I have a streaming pipeline the only solution I can think of currently is stopping my pipeline when date rolls over and then starting a new streaming pipeline with table name that has updated date. – PUG Aug 04 '17 at 16:17
  • @PUG have you tried `Write.toTableReference` in the newer versions? https://beam.apache.org/documentation/sdks/javadoc/0.4.0/index.html?org/apache/beam/sdk/io/gcp/bigquery/class-use/BigQueryIO.Write.Bound.html – Narek Aug 08 '17 at 04:16