Storing null values in avro files

Question

I have some json data that looks like this:

  {
    "id": 1998983092,
    "name": "Test Name 1",
    "type": "search string",
    "creationDate": "2017-06-06T13:49:15.091+0000",
    "lastModificationDate": "2017-06-28T14:53:19.698+0000",
    "lastModifiedUsername": "testuser@test.com",
    "lockedQuery": false,
    "lockedByUsername": null
  }

I am able to add the lockedQuery null value to a GenericRecord object without issue.

GenericRecord record = new GenericData.Record(schema);
if(json.isNull("lockedQuery")){
    record.put("lockedQuery", null);
}

However, later when I attempt to write that GenericRecord object to an avro file I get a null pointer exception.

File file = new File("~/test.arvo");
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);
dataFileWriter.create(schema, file);
for(GenericRecord record: masterList) {
    dataFileWriter.append(record); // NULL POINTER HERE
}

When I run that code I get the following exception. Any tips on how to process a null value into an Avro file much appreciated. Thanks in advance.

java.lang.NullPointerException: null of boolean in field lockedQuery of 
com.mydomain.test1.domain.MyAvroRecord
Exception in thread "main" java.lang.RuntimeException: 
org.apache.avro.file.DataFileWriter$AppendWriteException: 
java.lang.NullPointerException: null of boolean in field lockedQuery of 
com.mydomain.test1.domain.MyAvroRecord
at com.mydomain.avro.App.main(App.java:198)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: 
java.lang.NullPointerException: null of boolean in field lockedQuery of 
com.mydomain.test1.domain.MyAvroRecord
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)

EDIT: here is the MyAvroRecord

public class MyAvroRecord {
    long id;
    String name;
    String type;
    Date timestamp;
    Date lastModifcationDate;
    String lastModifiedUsername;
    Boolean lockedQuery;

It would probably help to have the definition of `MyAvroRecord`. I'm not particularly an Avro specialist, but in general in Java you cannot store `null` in a `boolean` (lower case) field, as it is a primitive. If you want to store a null in a boolean, you have to use `Boolean` (upper case), which is an Object and can be null. — ced-b, Aug 13 '17 at 16:34
Thanks for the suggestion. I added the member definitions from MyAvroRecord. The schema file matches the types. I was using boolean and switch to Boolean objects but still have the same null pointer. When I store "false" as a default value the error goes away but I still cannot use null with an object based Boolean. — mba12, Aug 13 '17 at 16:46

Vladimir Kroz · Accepted Answer · 2017-08-15T13:03:29.473

34

To be able to set Avro field to null you should allow this in Avro schema, by adding null as one of the possible types of the field. Take a look on example from Avro documentation:

{
  "type": "record",
  "name": "MyRecord",
  "fields" : [
    {"name": "userId", "type": "long"},              // mandatory field
    {"name": "userName", "type": ["null", "string"]} // optional field 
  ]
}

here userName is declared as composite type which could be either null or string. This kind of definition allows to set userName field to null. As contrast userId can only contain long values, hence attempt to set userId to null will result in NullPointerException.

edited Aug 15 '17 at 13:03

answered Aug 13 '17 at 16:51

Vladimir Kroz

5,237
6
39
50

2

How can it be done in Java interface without using schema file? – UsTa Apr 19 '18 at 11:47
5

You can do that using the SchemaBuilder by setting the type of the field as follows `.type(SchemaBuilder.unionOf().nullType().and().stringType().endUnion())` – panksdmz Jan 29 '19 at 15:11

score 7 · Answer 2 · edited Jun 14 '19 at 11:31

I have this issue too and now resolved it.

I found @Nullable annotation in Apache Avro to declare the field is nullable.

So, in this example, we should

import org.apache.avro.reflect.Nullable;

public class MyAvroRecord {
    long id;
    String name;
    String type;
    Date timestamp;
    Date lastModifcationDate;
    String lastModifiedUsername;
    @Nullable
    Boolean lockedQuery;
}

Storing null values in avro files

2 Answers2

Linked