Generate Avro Schema from certain Java Object

Question

Apache Avro provides a compact, fast, binary data format, rich data structure for serialization. However, it requires user to define a schema (in JSON) for object which need to be serialized.

In some case, this can not be possible (e.g: the class of that Java object has some members whose types are external java classes in external libraries). Hence, I wonder there is a tool can get the information from object's .class file and generate the Avro schema for that object (like Gson use object's .class information to convert certain object to JSON string).

Interesting question. There exists a tool which can generate JSON Schemas from Java classes (jsonschema2pojo) and I have a tool which can generate Avro schema from JSON Schemas (json-schema-avro). However, the former tool can only generate JSON Schema v3, and my tool awais JSON Schema v4 as inputs... — fge, Apr 10 '14 at 23:08
Thank you for your answer, do you mean that you wrote a tool which can convert Avro Schema from JSON schema? — Richard Le, Apr 11 '14 at 15:37

score 39 · Accepted Answer · edited May 23 '17 at 12:17

39

Take a look at the Java reflection API.

Getting a schema looks like:

Schema schema = ReflectData.get().getSchema(T);

See the example from Doug on another question for a working example.

Credits of this answer belong to Sean Busby.

edited May 23 '17 at 12:17

Community

1
1

answered Jul 01 '14 at 07:38

MoustafaAAtta

1,011
12
17

This works well with non-nullable columns, but I ve some fields that are nullable. Is there a way to make those fields nullable in Aro schema. Otherwise it throws an exception = org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: in models.RawData in double null of double in field offset of models.RawData – Remis Haroon - رامز Nov 01 '21 at 13:06
1

For nullable fields, use the AllowNull sub-class: `Schema schema = ReflectData.AllowNull.get().getSchema(T);` – Odysseus Mar 25 '22 at 22:34

score 11 · Answer 2 · answered Jan 10 '18 at 14:13

Here's how to Generate an Avro Schema from POJO definition

ObjectMapper mapper = new ObjectMapper(new AvroFactory());
AvroSchemaGenerator gen = new AvroSchemaGenerator();
mapper.acceptJsonFormatVisitor(RootType.class, gen);
AvroSchema schemaWrapper = gen.getGeneratedSchema();
org.apache.avro.Schema avroSchema = schemaWrapper.getAvroSchema();
String asJson = avroSchema.toString(true);

score 5 · Answer 3 · answered Apr 23 '19 at 10:16

** Example**

Pojo class

public class ExportData implements Serializable {
    private String body;
    // ... getters and setters
}

Serialize

File file = new File(fileName);
DatumWriter<ExportData> writer = new ReflectDatumWriter<>(ExportData.class);
DataFileWriter<ExportData> dataFileWriter = new DataFileWriter<>(writer);
Schema schema = ReflectData.get().getSchema(ExportData.class);
dataFileWriter.create(schema, file);
for (Row row : resultSet) {
    String rec = row.getString(0);
    dataFileWriter.append(new ExportData(rec));
}
dataFileWriter.close();

Deserialize

File file = new File(avroFilePath);
DatumReader<ExportData> datumReader = new ReflectDatumReader<>(ExportData.class);
DataFileReader<ExportData> dataFileReader = new DataFileReader<>(file, datumReader);
ExportData record = null;
while (dataFileReader.hasNext()){
    record = dataFileReader.next(record);
    // process record
}

This works well with non-nullable columns, but I ve some fields that are nullable. Is there a way to make those fields nullable in Aro schema. Otherwise it throws an exception = org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: in models.RawData in double null of double in field offset of models.RawData — Remis Haroon - رامز, Nov 01 '21 at 13:06

Generate Avro Schema from certain Java Object

3 Answers3

Linked