15

I get Message size too large exception, when I try to send a message which is over 1 Mb size. The error appears in my client application, when I try to produce a message. After a little googling I found out that the settings should be changed in order to increase max message size. Well, I did that in /kafka/config/server.properties file. I added next 2 settings:

message.max.bytes=15728640
replica.fetch.max.bytes=15728640

Also, I added fetch.message.max.bytes=15728640 to the /kafka/config/consumer.properties file. All other settings remain default.

I did kafka server restart, but I'm still getting the same error.

P.S Kafka version is 1.1.0.

Nitin
  • 3,533
  • 2
  • 26
  • 36
managerger
  • 728
  • 1
  • 10
  • 31
  • Did you look at this question in this [link](https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb)? – c.guzel Dec 13 '19 at 13:06
  • Do you refer to some specific point from the mentioned link? Because, I did configuration exactly like stated there. Am I missed something? – managerger Dec 13 '19 at 15:48

2 Answers2

18

You have the right configuration however you need to also set max.request.size on the producer side.

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);

max.request.size The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size.

On the Broker side, you have already configured the below parameter that should work

message.max.bytes The largest record batch size allowed by Kafka.

replica.fetch.max.bytes The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

On the topic side max.message.bytes which is not required in case you have already set message.max.bytes in the broker side

max.message.bytes - this is the largest size of the message the broker will allow being appended to the topic. This size is validated pre-compression. (Defaults to broker's message.max.bytes.)

Refrence https://kafka.apache.org/documentation/

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Nitin
  • 3,533
  • 2
  • 26
  • 36
  • 3
    Setting message.max.bytes for producer on the client side does a trick. Thank you, sir! – managerger Dec 14 '19 at 10:44
  • I had to set max.message.bytes on the topic level – Juri Aug 12 '22 at 17:44
  • In my case, using `confluent_kafka`, which is based on `librdkafka`, the producer needs to set the config: `'message.max.bytes` instead. HTH Reference: https://github.com/confluentinc/confluent-kafka-python/issues/325 – xmar Jun 06 '23 at 13:09
0

I think you might need to set batch.size too.

I got a similar problem. I created a topic with max.message.bytes=4096, and I used kafka-producer-perf-test.sh for testing. I also set max.request.size=4096, but it seems does work properly.

If I go with just one message, it's fine.

./kafka/bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 max.request.size=4096 --record-size 4008 --topic test-4K-1RF-topic-0 --num-records 1 --throughput 1

If I go with just more than one message, I will get MESSAGE_TOO_LARGE error.

./kafka/bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 max.request.size=4096 --record-size 4008 --topic test-4K-1RF-topic-0 --num-records 2 --throughput 1

So I guess it is about batch size. Then, I found batch.size

so the below snippet is working now!

./kafka/bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 batch.size=4096 --record-size 4008 --topic test-4K-1RF-topic-0 --num-records 200 --throughput 10

Edited: I think max.request.size is to limit the size of a package that a producer sent to a Kafka broker. A package can consist of multiple batches. The batch size is limited by batch.size.

JengdiB
  • 56
  • 3