I have one of our Python systems generating Parquet files using Pandas and fastparquet. These are to be read by a Scala system that runs atop Akka streams.
Akka does provide a source for reading Avro Parquet files. However, when I try to read the file, I end up with
java.lang.IllegalArgumentException: INT96 not yet implemented.
This is one of the columns that does not need to be read for the Scala application to work. My question is whether I can specify a schema and get just that one column out considering that the generated file is from fastparquet.
The relevant snippet which generates a source for reading Parquet files is:
.map(result => {
val path = s"s3a://${result.bucketName}/${result.key}"
val file = HadoopInputFile.fromPath(new Path(path), hadoopConfig)
val reader: ParquetReader[GenericRecord] =
AvroParquetReader
.builder[GenericRecord](file)
.withConf(hadoopConfig)
.build()
AvroParquetSource(reader)
})