6

In which file I can increase the maximum message size for Kafka? (I'm sending records of String, byte[] and when I send byte[] of 770kb Kafka does not send the message)

neb
  • 135
  • 1
  • 3
  • 9
  • This q&a probably explains it completely: https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb/39026744 - it's for Kafka 0.8 but I believe this is still correct for newer versions of Kafka. Let us know if this works, then I can mark it as a duplicate. (Which is good, it just points other people with the same concern to the right question) – Erwin Bolwidt May 07 '18 at 06:02
  • @ErwinBolwidt I set replica.fetch.max.bytes to 1000000 and max.request.size to 1000000 in producer and max.partition.fetch.bytes to 1000000 in comsumer but still the message isn't sent – neb May 07 '18 at 06:47
  • 1
    And `message.max.bytes` ? – Erwin Bolwidt May 07 '18 at 07:29
  • duplicate of https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb – technogeek1995 Mar 15 '19 at 18:33
  • 1
    Possible duplicate of [How can I send large messages with Kafka (over 15MB)?](https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb) – technogeek1995 Mar 15 '19 at 18:33

1 Answers1

9

In kafka 0.11.0 following four config options need to be set

Broker config options (details) :

  1. message.max.bytes - The largest record batch size allowed by Kafka.

  2. replica.fetch.max.bytes - The number of bytes of messages to attempt to fetch for each partition.

Producer config options (details) :

  1. max.request.size - The maximum size of a request in bytes. It is advisable to match this value with (message.max.bytes).

Consumer config options (details) :

  1. max.partition.fetch.bytes - max number of bytes per partition returned by the server. should be larger than the max.message.size so consumer can read the largest message sent by the broker.
Swapnil
  • 247
  • 1
  • 7