3

I have the following Terraform configuration to create an AWS Athena Iceberg table:

resource "aws_glue_catalog_table" "my_table" {
  name          = "my_table"
  database_name = "my_db"

  table_type = "TABLE"

  parameters = {
    "table_type" = "iceberg"
  }

  storage_descriptor {
    location      = "s3://my_data/my_table"
    input_format  = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"
    output_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"

    ser_de_info {
      name                  = "my-table-stream"
      serialization_library = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"

      parameters = {
        "serialization.format" = 1
      }
    }

    columns {
      name = "some_date"
      type = "date"
    }

    columns {
      name = "some_name"
      type = "string"
    }

    columns {
      name    = "some_count"
      type    = "bigint"
    }
  }
}

when that is run the table is created as expected. However when I run select * from my_table from the Athena Query Editor I get the following error:

GENERIC_USER_ERROR: Detected Iceberg type table without metadata location. Please make sure an Iceberg-enabled compute engine such as Athena or EMR Spark is used to create the table, or the table is created by using the Iceberg open source library. Setting table_type parameter in Glue metastore to create an Iceberg table is not supported.

How to properly configure an Athena Iceberg table with Terraform?

Randomize
  • 8,651
  • 18
  • 78
  • 133
  • Same question as https://stackoverflow.com/questions/75581933/how-to-deploy-iceberg-tables-to-aws-through-terraform i guess. Did you find any workaround? – pauetpupa Mar 06 '23 at 16:08
  • No, I have took it out from TF and created the tables programmatically. – Randomize Mar 07 '23 at 05:53
  • Programmatically calling Athena query I guess, right? With glue sdk you would have the same problem of not creating the `.metadata.json` if I'm not wrong. – pauetpupa Mar 07 '23 at 11:58
  • Yes, using Athena (via python/boto3). Not sure about glue part as I have opened another related question https://stackoverflow.com/questions/75411633/how-to-use-column-names-with-spaces-with-aws-athena-iceberg-tables – Randomize Mar 07 '23 at 12:01
  • The problem with athena queries I guess that it's hard to create a good tool to maintain, evolve, migrate and align in multiple environments, ... the "SQL" table definitions. I hope in some future, the hashicorp provider allows iceberg tables – pauetpupa Mar 07 '23 at 12:32

0 Answers0