Is it possible to have an optional field in an Avro schema (i.e. the field does not appear at all in the .json file)?

Question

Is it possible to have an optional field in an Avro schema (i.e. the field does not appear at all in the .JSON file)?

In my Avro schema, I have two fields:

{"name": "author", "type": ["null", "string"], "default": null},
{"name": "importance", "type": ["null", "string"], "default": null},

And in my JSON files those two fields can exist or not.

However, when they do not exist, I receive an error (e.g. when I test such a JSON file using avro-tools command line client):

Expected field name not found: author

I understand that as long as the field name exists in a JSON, it can be null, or a string value, but what I'm trying to express is something like "this JSON is valid if the those field names do not exist, OR if they exist and they are null or string".

Is this possible to express in an Avro schema? If so, how?

@red1ynx unfortunately no. I still can't express a totally optional JSON field using Avro schema. — Emre Sevinç, May 19 '15 at 13:26
I've struggled with this too. See https://stackoverflow.com/questions/45193729/avro-how-can-i-use-default-fields-when-i-dont-know-the-exact-schema-that-the — JohnK, Jul 28 '17 at 14:20
you may want to add some code on how you are parsing from json to avro — hlagos, Aug 22 '19 at 12:56

score 47 · Answer 1 · edited Jan 27 '23 at 13:26

47

you can define the default attribute as undefined example. so the field can be skipped.

{ 
   "name": "first_name",
   "type": "string",
   "default": "undefined"
},

Also all field are manadatory in avro. if you want it to be optional, then union its type with null. example:

{
    "name": "username",
    "type": [
      "null",
      "string"
    ],
    "default": null
},

edited Jan 27 '23 at 13:26

Banana

2,435
7
34
60

answered Aug 21 '19 at 06:17

arvin_v_s

1,036
1
12
18

10

Isn't OP using exactly the same as in this answer? – Deqing Mar 09 '22 at 03:19

score 15 · Answer 2 · answered Sep 05 '18 at 20:02

According to avro specification this is possible, using the default attribute.

See https://avro.apache.org/docs/1.8.2/spec.html

default: A default value for this field, used when reading instances that lack this field (optional). Permitted values depend on the field's schema type, according to the table below. Default values for union fields correspond to the first schema in the union.

At the example you gave, you do add the default attribute with value "null", so this should work. However, supporting this depends also on the library you use for reading the avro message (there are libraries at c,c++,python,java,c#,ruby etc.). Maybe (probably) the library you use lack this feature.

Is it possible to have an optional field in an Avro schema (i.e. the field does not appear at all in the .json file)?

2 Answers2

Linked

Related