27

How do you extract first the schema and then the data from an avro file in Java? Identical to this question except in java.

I've seen examples of how to get the schema from an avsc file but not an avro file. What direction should I be looking in?

Schema schema = new Schema.Parser().parse(
    new File("/home/Hadoop/Avro/schema/emp.avsc")
);
halfer
  • 19,824
  • 17
  • 99
  • 186
mba12
  • 2,702
  • 6
  • 37
  • 56

3 Answers3

41

If you want know the schema of a Avro file without having to generate the corresponding classes or care about which class the file belongs to, you can use the GenericDatumReader:

DatumReader<GenericRecord> datumReader = new GenericDatumReader<>();
DataFileReader<GenericRecord> dataFileReader = new DataFileReader<>(new File("file.avro"), datumReader);
Schema schema = dataFileReader.getSchema();
System.out.println(schema);

And then you can read the data inside the file:

GenericRecord record = null;
while (dataFileReader.hasNext()) {
    record = dataFileReader.next(record);
    System.out.println(record);
}
Helder Pereira
  • 5,522
  • 2
  • 35
  • 52
  • 2
    For those using the C# Avro Apache library, the utility function `DataFileReader.OpenReader(filename);` can be used to instantiate the `dataFileReader`. Once instantiated, it the dataFileReader is used just like in Java. – Berthier Lemieux Sep 16 '18 at 10:54
  • 1
    I am trying to read the schema and data from a byte array instead of a File (containing both schema and payload). How can I do it? – Alberto Castaño Mar 06 '20 at 08:03
2

Thanks for @Helder Pereira's answer. As a complement, the schema can also be fetched from getSchema() of GenericRecord instance.
Here is an live demo about it, the link above shows how to get data and schema in java for Parquet, ORC and AVRO data format.

Eugene
  • 10,627
  • 5
  • 49
  • 67
1

You can use the data bricks library as shown here https://github.com/databricks/spark-avro which will load the avro file into a Dataframe (Dataset<Row>)

Once you have a Dataset<Row>, you can directly get the schema using df.schema()

Carlos Bribiescas
  • 4,197
  • 9
  • 35
  • 66
  • 3
    Apologies, I just realized you weren't actually using Spark to begin with. If you're not already using Spark, then my solution is more trouble than its worth. I'll leave the answer though, in case someone coming from the Spark perspective has the same question – Carlos Bribiescas Aug 09 '17 at 22:19
  • I am not using Spark. Just the plain vanilla avro tools jar but thank you. – mba12 Aug 09 '17 at 23:22