Kafka Producer limits producer messages to 1024 bytes

Question

I am trying to produce kafka message using the terminal. I downloaded this version kafka_2.12-2.4.1.tgz. I tried to start the Zookeeper server and then the the Kafka server. I have a json which is more than 1024 bytes. It's not allowed to send more than 1024 bytes.

I tried to refer to How can I send large messages with Kafka (over 15MB)?. But that didn't help. Not sure if I am missing any config change.

Producer Config

max.request.size=2147483647 
socket.buffer.size=2147483647

Server Config

socket.request.max.bytes=2147483647
replica.fetch.max.bytes=2147483647
message.max.bytes=2147483647
max.message.bytes=2147483647
replica.fetch.max.bytes=2147483647

Consumer Config

max.request.size=2147483647
fetch.message.max.bytes=2147483647
max.partition.fetch.bytes=2147483647

provide your producer configuration – bottaio Mar 29 '20 at 16:53 — bottaio, Mar 29 '20 at 16:53
@bottaio : Added the configs – codinginprogress Mar 30 '20 at 00:50 — codinginprogress, Mar 30 '20 at 00:50

score 3 · Answer 1 · edited Mar 03 '22 at 12:51

You should increase max.partition.fetch.bytes consumer config parameter too for receiving large messages.

max.partition.fetch.bytes: The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.

Kafka Producer limits producer messages to 1024 bytes

1 Answers1

Linked