Not able to consume large messages with Kafka (over 1.3 MB)?

Question

I followed the link How can I send large messages with Kafka (over 15MB)? to resolve the kafka msg limit issue. But no luck

I tried increasing

A.) On Broker:

message.max.bytes=15728640 
replica.fetch.max.bytes=15728640

B.) On Consumer: fetch.message.max.bytes=15728640

Still facing the same problem. Not able to consume data that is over 1.3 MB on a particular topic

In my application, a msg in sent on a topic from python code and is consumed on node server.

Were you able to send them without error, and you just cannot consume them? — OneCricketeer, Aug 17 '18 at 23:45
I am able to send and getting problem at consuming the message — AaJ, Aug 20 '18 at 17:07

score 2 · Answer 1 · edited Aug 20 '18 at 17:11

Kafka does have strict restriction over the size of data; default is 1MB. I believe you have missed topic level config.

There are multiple configs at different levels:

You have a broker setting message.max.bytes (default is 1000012) http://kafka.apache.org/documentation/#brokerconfigs

There is a topic level config max.message.bytes (default is 1000012) http://kafka.apache.org/documentation/#topicconfigs

Consumer has max.partition.fetch.bytes (default is 1048576) http://kafka.apache.org/documentation/#consumerconfigs

1 Answers1