Handle nested Avro record when the one is null

Question

I've got a question about nested Avro records. Actually, I convert a Json file to an Avro one.

The Json file is parsed using Jackson.

The thing is:

the Avro schema contains all the possible fields (including nested ones) the Json could contain and sometimes one or several nested Json are missing.
when Jackson parses a Json file, if a nested Json is missing, the reference of the relative jackson object is null.
when I try creating the Avro record, I got an exception as the nested Avro record cannot be created from a null reference.

My question is the following: how handling null nested records? I use default value for fields, is there something similar for record? If not, except creating the nested Json manually if not present (with empty values), is there another way to process?

BRgds, Eddy.

Little more explanation about what is going on here https://stackoverflow.com/questions/27485580/how-to-fix-expected-start-union-got-value-number-int-when-converting-json-to-av — Michael, Feb 23 '18 at 03:56

EddyA · Answer 1 · 2017-10-17T12:55:17.397

I found the problem, it was nothing about what I explained before, my bad ^^!

In fact there exists a kind of bug in Avro ... to be concised, I filled the nested avro record created with a builder and set all the fields:

deviceInfoBuilder = MobtabAvroDeviceInfo.newBuilder();
deviceInfoBuilder.setModelName(mobtabJsonHit.getMobtabJsonDeviceInfo().getModuleName());
deviceInfoBuilder.setOperatingSystemName(mobtabJsonHit.getMobtabJsonDeviceInfo().getOperatingSystemName());
deviceInfoBuilder.setOperatingSystemVersion(mobtabJsonHit.getMobtabJsonDeviceInfo().getOperatingSystemVersion());
deviceInfoBuilder.setType(mobtabJsonHit.getMobtabJsonDeviceInfo().getDeviceType());
deviceInfoBuilder.setAdId(mobtabJsonHit.getMobtabJsonDeviceInfo().getDeviceAdId());

When the information wasn't available (i.e. the nested Jackson was not present), I created an "empty" nested Avro record and provided it to the builder as follows:

private final static MobtabAvroDeviceInfo emptyMobtabAvroDeviceInfo =
new MobtabAvroDeviceInfo(null, null, null, null, null);
deviceInfoBuilder = MobtabAvroDeviceInfo.newBuilder(emptyMobtabAvroDeviceInfo);

And then, I got the following exception:

java.lang.NullPointerException: null of string of fr.mediametrie.internet.pame.meter.avro.MobtabAvroDeviceInfo of fr.mediametrie.internet.pame.meter.avro.MobtabAvroHit

Although my Avro schema enables "null" values ...

{
    "name": "device_info",
    "type": {
        "type": "record",
        "name": "MobtabAvroDeviceInfo",
        "fields": [
            {
                "name": "model_name",
                "type": ["null", "string"],
                "default": null
            },
            {
                "name": "operating_system_name",
                "type": ["null", "string"],
                "default": null
            },
            {
                "name": "operating_system_version",
                "type": ["null", "string"],
                "default": null
            },
            {
                "name": "type",
                "type": ["null", "string"],
                "default": null
            },
            {
                "name": "ad_id",
                "type": ["null", "string"],
                "default": null
            }
        ]
    }
}

To solve the problem, I've just created a new builder without specifying an "empty" nested Avro record in parameter:

deviceInfoBuilder = MobtabAvroDeviceInfo.newBuilder();

To conclude, it seems I cannot create an "empty" nested Avro using default values and provide the builder with but I must let the builder building that object using the default values specified in the schema. Strange, but it works now :)

score 0 · Answer 2 · answered Nov 08 '17 at 12:17

I finally catched the problem ... I got two different schemas (I only post one in this post), both containing a subrecord called mobile_web_event ... When generating the avro sources, the subrecord B erased the A and, as the second one has different default values, I thought thetre was a bug in the avro source generation ... ^^!

Handle nested Avro record when the one is null

2 Answers2