JSON to Avro Decoding - AvroTypeException: Expected field name not found

Question

I am using Avro 1.8.2 and I am trying to convert JSON into a GenericRecord

 DatumReader<GenericData.Record> datumReader = new GenericDatumReader<>(schema);
 Decoder decoder = DecoderFactory.get().jsonDecoder(schema, jsonStr);
 datumReader.read(null, decoder)

I get JSON data from a third party and I have no control over the elements. The AVRO schema is

{
    "namespace":"com.avro.generated",
    "type":"record",
    "name":"TestEvent",
    "fields":[
        {"name":"userId","type":"string"},
        {"name":"frm","type":"string"},
        {"name":"issuerName","type":"string"},
        {"name":"profileId","type":"string"}
    ]
}

If I use this JSON

{
    "userId":"5435tert34tgcb21391f7bda71",
    "frm":"somerm",
    "issuerName":"somenameorts",
    "profileId":"0werwerwer0000-0000-000000000000"
}

It works fine. However if the json does not contain the frm element as shown below

{
    "userId":"5435tert34tgcb21391f7bda71",
    "issuerName":"somenameorts",
    "profileId":"0werwerwer0000-0000-000000000000"
}

Then I get this exception

org.apache.avro.AvroTypeException: Expected field name not found: frm.

Is there any way to make this work?. I have no control over the JSON. I have read other SO posts about using schemas like

{"name":"frm","type":["null","string"],"default": "null"}

But none of this is working

Thanks

The only way I am able to make this work is by converting it into the actual generated class which extends SpecificRecordBase. Something like this — Jeenson Ephraim, Aug 04 '20 at 00:24

score 1 · Answer 1 · answered Jul 30 '20 at 15:29

1

All fields are mandatory in AVRO, but you can provide a default so that it has the field.

{
    "namespace":"com.avro.generated",
    "type":"record",
    "name":"TestEvent",
    "fields":[
        {"name":"userId","type":["null","string"], "default": null},
        {"name":"frm","type":["null","string"], "default": null},
        {"name":"issuerName","type":["null","string"], "default": null},
        {"name":"profileId","type":["null","string"], "default": null}
    ]
}

EDIT: sorry didn't read the end of your message. What is the error when you say it is not working (also, notice that null is not to be quoted)

answered Jul 30 '20 at 15:29

BenoitParis

3,166
4
29
56

@BenoliParis Evene with the working complete JSON, I get this error. org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING – Jeenson Ephraim Jul 30 '20 at 21:07
@JeensonEphraim Seems like the next bug, which might be: https://stackoverflow.com/questions/27485580/how-to-fix-expected-start-union-got-value-number-int-when-converting-json-to-av – BenoitParis Jul 31 '20 at 08:52

score 0 · Answer 2 · answered Aug 04 '20 at 00:30

The only way I am able to make this work is by converting it into the actual generated class which extends SpecificRecordBase and the converting it back into a GenericRecord. I dont know if this is a valid way of doing it , but just posting it

ObjectMapper mapper = new ObjectMapper();
TestEvent node = mapper.readValue(jsonStr.getBytes(), TestEvent.class);
// This gives the actual Class

I can convert this back to a generic record using

Schema schema = TestEvent.getClassSchema();
ReflectDatumWriter datumWriter = new ReflectDatumWriter<>(schema);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();

BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null);
datumWriter.write(node, encoder);
encoder.flush();

DatumReader<GenericRecord> datumReader = new GenericDatumReader<>(schema);
BinaryDecoder  decoder = DecoderFactory.get().binaryDecoder(outputStream.toByteArray(), null);
datumReader.read(null, decoder)

JSON to Avro Decoding - AvroTypeException: Expected field name not found

2 Answers2