How to send a list of objects using avro & kafka

Question

I'm trying to send a list of objects using Avro & Kafka, my main problem is that my list size may change frequently, so I don't know how to build a dynamic Avro schema since as far as I understand Avro schema meant for well-known structure.

Anybody knows how to do that?

Avro support some complex types like arras, maps. This give you some flexibility: http://stackoverflow.com/questions/39232395/kafka-kstreams-processing-timeouts/39237089#39237089 — Matthias J. Sax, Dec 15 '16 at 17:33

score 1 · Answer 1 · answered Dec 17 '16 at 08:38

I think that the easiest way is to make a class with list, for example:

public class AvroObj {

    private List<TestObj> list;

    public List<TestObj> getList() {
        return list;
    }

    public void setList(List<TestObj> list) {
        this.list = list;
    }
}

create a schema:

Schema schema = ReflectData.get().getSchema(AvroObj.class);

serialize it to bytes using ReflectDatumWriter (i offer it cause with another datum writers you may catch ClassCastException)

try(ByteArrayOutputStream out = new ByteArrayOutputStream()) {
            BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
            DatumWriter<AvroObj> writer = new ReflectDatumWriter<>(schema);
            writer.write(avroObj, encoder);
            encoder.flush();
            byte[] bytes = out.toByteArray();
        }

then send bytes with kafka producer.

deserialise bytes that consumer received:

DatumReader<AvroObj> reader1 = new ReflectDatumReader<AvroObj>(schema);
        Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
        AvroObj decodedAvroObj = reader1.read(null, decoder);

How to send a list of objects using avro & kafka

1 Answers1