0

I need the producer to serialize and send a custom object that contains a Bytebuffer (video chunk) and a String:

public class NvrWSBinaryMessage {
  private String messageUuid;
  private final ByteBuffer payload;
...

And have the consumer correctly deserialize.

I've not found a correct way to create a custom serializer/deserializer for the object having the ByteBuffer. I tried to send the object as a bytearray without any success.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Jocheinfa
  • 25
  • 1
  • 4
  • I'm 99.% sure you have to store a `byte[]` on the message rather than a `ByteBuffer`. I assume you are using Avro, as this could be different if you are serialising in a different format. – Augusto Oct 23 '21 at 20:33
  • Thanks Augusto, i'm not using Avro, should I?. Also if I change a bytbuffer for a bytearray, Should I serialize the whole object to byte array even if the object has a byte array? – Jocheinfa Oct 23 '21 at 22:51
  • 1
    You don't need to use Avro, no, but it'll let you send data without having to write a custom serializer – OneCricketeer Oct 24 '21 at 13:12

1 Answers1

0

There is a ByteBufferSerializer / ByteBufferDeserializer you could use for your kafka clients (producer/consumer), however, that would require you to send the ByteBuffer payload as the kafka message/record value and String messageUuid as the key. serialization 2.8 docs

If you don't have this flexibility, then I'd suggest leveraging jackson annotations (similiar to reference) on the object you want to send as the record value, and using the JsonSerializer / JsonDeserializer for your clients.

Serialization will convert an object into byte code format, so it gets tricky when passing an object holding bytes as a value to be serialized without any extra configuration. The annotations are used to remove any serialization ambiguity, allowing the Kafka provided serializers to do the job without issue.

Nerm
  • 170
  • 2
  • 11
  • Json doesn't have a bytes type, though, so are you suggesting base64 encoding? Why not use Avro or Protobuf? – OneCricketeer Oct 24 '21 at 13:11
  • Yes - a string of 0's and 1's shouldn't cause too much havoc to a project. If you're curious, check out that link. There isn't an issue with Avro/Protobuf, it could definitely be used to solve this problem. However, it brings a layer of complexity, one that might not be necessary to solve the problem. – Nerm Oct 24 '21 at 18:13