1

I wanted to add a new field into an AVRO schema of type "record" that cannot be null and therefore has a default value. The topic is set to compatibility type "Full_Transitive".

The schema did not change from the last version, only the last field produktType was added:

{
  "type": "record",
  "name": "Finished",
  "namespace": "com.domain.finishing",
  "doc": "Schema to indicate the end of the ongoing saga...",
  "fields": [
    {
      "name": "numberOfAThing",
      "type": [
        "null",
        {
          "type": "string",
          "avro.java.string": "String"
        }
      ],
      "default": null
    },
    {
      "name": "previousNumbersOfThings",
      "type": {
        "type": "array",
        "items": {
          "type": "string",
          "avro.java.string": "String"
        }
      },
      "default": []
    },
    {
      "name": "produktType",
      "type": {
        "type": "record",
        "name": "ProduktType",
        "fields": [
          {
            "name": "art",
            "type": "int",
            "default": 1
          },
          {
            "name": "code",
            "type": "int",
            "default": 10003
          }
        ]
      },
      "default": { "art": 1, "code": 10003 }
    }
  ]
}

I've checked with the schema-registry that the new version of the schema is compatible.

But when we try to read old messages that do not contain that new field with the new schema (where the defaults are) there is a EOF Exception and it does not seem to work.

The part that causes headaches is the new added field "produktType". It cannot be null so we tried adding defaults. Which is possible for primitive type fields ("int" and so on). The line "default": { "art": 1, "code": 10003 } seems to be ok with the schema-registry but does not seem to have an effect when we read messages from the topic that do not contain this field.

The schema registry also marks it as not compatible when the last "default": { "art": 1, "code": 10003 } line is missing (but also "default": true works regarding schema compatibility...).

The AVRO specification for complex types contains an example for type "record" and default {"a": 1} so that is where we got that idea from. But since its not working something is still wrong.

There are similar questions like this one claiming records can only have null as a default or this un-answered one.

Is this supposed to work? And if so how can defaults for these "type": "record" fields be defined? Or is it still true that records can only have null as default?

Thanks!

Update on the compatibility cases:

Schema V1 (old one without the new field): can read v1 and v2 records. Schema V2 (new field added): cannot read v1 records, can read v2 records

The case where a consumer using schema v2 encountering records using v1 is the surprising one - as I thought the defaults are for that purpose.

Even weirder: when I don't set the new field values at all. The v2 record does contain some values:

good art field, bad code field

I have no idea where the value for code is from. The schema uses other numbers for its defaults: trying default values for records

So one of them seems to work, the other does not.

wemu
  • 7,952
  • 4
  • 30
  • 59
  • 1
    Can you provide examples of generating the data and verifying the fields are/aren't there? – OneCricketeer Dec 02 '21 at 22:02
  • @OneCricketeer thanks for your reply! I've tried to add all the combinations of schemas / cases and what they do. The fields are there if not set (using schema v2 and creating a record without the produktArt field (was produktType), but only one of them is using the default value, the other ("code") has a very wrong value. – wemu Dec 03 '21 at 12:55
  • can you post your code for actually decoding records? i have a suspicion you are trying to use the new schema as the WRITER schema argument (where the original should go) and not just as the reader schema argument – radai Mar 28 '22 at 17:43
  • sorry I no longer work in that project. The avro setup was done purely through the kafka API. So not sure if that was done wrong :/ – wemu Mar 28 '22 at 19:39

0 Answers0