0

How would you describe spark StructType data type in an avro schema? I am generating a parquet file, the format of which is described in an avro schema. This file is then loaded from S3 into spark. There is an array and map data types but these do not correspond to the StructType.

naimdjon
  • 3,162
  • 1
  • 20
  • 41

1 Answers1

1

Using the package org.apache.spark.sql.avro (Spark 2.4) you can convert sparkSQL schemas to avro schemas and viceversa.

You cant try this way:

import org.apache.spark.sql.avro.SchemaConverters val sqlType = SchemaConverters.toSqlType(avroSchema) var rowRDD = yourGeneircRecordRDD.map(record => genericRecordToRow(record, sqlType)) val df = sqlContext.createDataFrame(rowRDD , sqlType.dataType.asInstanceOf[StructType])

Here you can find more answers too: Code

mtapia
  • 11
  • 3