Recomended message length in a kafka topic

Question

I have a List of ids, with more or lest 400.000 ids, i need send to kafka the ids, i don't know if the best option is send the message split in n messages with x transactions, or if is better in one message adjusting like said in this post:

How can I send large messages with Kafka (over 15MB)?

Answer 1

This is a very generic question and it depends on how you want to process it. If your consumer is capable of processing each of id entries quickly, then you can put a lot of them into a single message. OTOH, if the processing is slow, it's better to publish more messages (across multiple partitions), so if you use consumer groups you wouldn't get group membership loss events etc.

Not to forget, there's also a limit on message size (as you've linked) with default of around 1mb.

In other words - you might need to perf-test on your own side, as it's hard to make a decision with only so little data.