1

I am new to kafka and have tried methods present online , like putting property compression.type="snappy"(even gzip and lz4). compression.codec=snappy . Still it's not working. I tried to put linger.ms = 5000 and then the compression type. But nothing of above is working.The message size is not getting compressed.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Preena
  • 11
  • 2
  • can you add more detail about what tool and how do you check the size of the produced messages? – Tuyen Luong Apr 06 '20 at 07:11
  • I had added logger to check the size . The exception is thrown that the size of the message has exceeded. Its size is 1mb – Preena Apr 06 '20 at 09:47
  • By default, kafka producer check the uncompressed message size against max.request.size even if compression.type is defined. Possible duplicated of https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb – Tuyen Luong Apr 06 '20 at 10:10
  • You can try increase Producer config max.request.size, if the compressed message is still greater than message.max.bytes then you have to increase broker level configs – Tuyen Luong Apr 06 '20 at 10:11
  • Does this answer your question? [How can I send large messages with Kafka (over 15MB)?](https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb) – Tuyen Luong Apr 06 '20 at 10:12

0 Answers0