Is there a better way to grab a list of Avro records from an InputStream?

Question

I have a ByteArrayInputStream that was serialized with a List<TestAvroModel> which is an implementation of SpecificRecord. I could not find a way for Avro to know about the list that was serialized so I did a hackish way to loop through the ByteArrayInputStream.

//TestAvroModel is an implementation of SpecificRecord
List<TestAvroModel> models;
ByteArrayOutputStream byteArrayStream = new ByteArrayOutputStream();
for(TestAvroModel model: models) {
    DatumWriter<SpecificRecord> writer = new SpecificDatumWriter<>(model.getSchema());
    Encoder encoder = new EncoderFactory().binaryEncoder(byteArrayStream, null);

    writer.write(model, encoder);
    encoder.flush();
}

//This was pre-serialized with a List of TestAvroModel
ByteArrayInputStream inputStream;

DatumReader<TestAvroModel> reader = new SpecificDatumReader<>(TestAvroModel.getClassSchema());
Decoder decoder = DecoderFactory().get().binaryDecoder(inputStream, null);

List<TestAvroModel> records = new ArrayList<>();
boolean eof = false;
while(!eof) {
    try {
        records.add(reader.read(null, decoder));
    catch(EOFException ex) {
        eof = true;
    }
}

This way worked and read the serialized List<TestAvroModel> one at a time and added it to my records list. Though looping through a DatumReader until an EOFException does not seem like the best way, but I haven't found a better way.

I couldn't find anything in the Avro libraries that dealt with an InputStream with multiple Avro records in it. Though it has to have breaking points in the stream for Avro to be able to read individual records the way I did above. To reiterate, does anyone know a better way to loop through the DatumReader then the way shown above?

I am curious how you serialized your data, can you share it with us? what about create a schema on the fly with a list of your records to handle this ? — hlagos, Oct 24 '18 at 12:33
@hlagos I have updated my question with a piece of the code on how I serialize my test data. Also I am confused with what you mean to create a schema on the fly for the list of records. — Aaron Waner, Oct 24 '18 at 18:56

score 0 · Answer 1 · answered Nov 10 '20 at 12:58

Decoder seems to define isEnd(): Boolean for that purpose:

Returns true if the current BinaryDecoder is at the end of its source data and cannot read any further without throwing an EOFException or other IOException.

This should work:

...
while(!decoder.isEnd()) {
  records.add(reader.read(null, decoder));
}

Is there a better way to grab a list of Avro records from an InputStream?

1 Answers1