RecordTooLargeException in kafka

Question

the following is kafka publishing code which is giving RecordTooLargeException exception.

tried all possible solutions given in stackoverflow giving info about different properties like max.request.size etc. but nothing worked. exact stack trace is

Caused by: org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 1696090 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

    @SuppressWarnings("unchecked")
    @Override
    public void run(String... args) throws Exception {

        JSONArray array = new JSONArray();

        for (int i = 0; i < 8000; i++) {
            JSONObject object = new JSONObject();
            object.put("no", 1);
            object.put("name", "Kella Vivek");
            object.put("salary", 1000);
            object.put("address", "2-143");
            object.put("city", "gpm");
            object.put("pin", 534316);
            object.put("dist", "west");
            object.put("state", "ap");
            object.put("username", "mff");
            object.put("password", "mff");
            array.add(object);
        }

        ObjectMapper mapper = new ObjectMapper();
        String string = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(array);

        template.send("consume", string);

    }

Check [this](http://www.alternatestack.com/development/app-development/kafka-message-size-issue-record-too-long-to-send) out. Hope it helps — Prathap Reddy, Aug 21 '20 at 19:06
This is general Apache Kafka question. There is just nothing with Spring to do. — Artem Bilan, Aug 21 '20 at 19:15
See here: https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-kafka-extra-props. But again: the "too large" problem is out of Spring scope. — Artem Bilan, Aug 21 '20 at 19:46
Does this answer your question? [How can I send large messages with Kafka (over 15MB)?](https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb) — OneCricketeer, Aug 20 '21 at 15:30

score 2 · Answer 1 · answered May 25 '21 at 09:20

This is not a spring problem. You need to tweak a number of parameters in kafka producer to make this working.

Now to answer your question I did the following to enable sending 100 mb messages.

Create the properties and set size for buffer.memory, message.max.bytes and max.request.size as per your requirements.

Properties producerProperties = new Properties();
producerProperties.put("buffer.memory", 104857600);
producerProperties.put("message.max.bytes", 104857600);
producerProperties.put("max.request.size", 104857600);
producerProperties.put("bootstrap.servers", kafkaBootstrapServers);
producerProperties.put("acks", "all");
producerProperties.put("retries", 0);
producerProperties.put("batch.size", 16384);
producerProperties.put("linger.ms", 1);
producerProperties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
producerProperties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Create Producer using the above properties:

KafkaProducer<String, String> producer = new KafkaProducer<>(producerProperties);

And now send.

private static void sendKafkaMessage(String payload,
         KafkaProducer<String, String> producer,
         String topic)
{
    logger.info("Sending Kafka message: " + payload);
    producer.send(new ProducerRecord<>(topic, payload));
}

You also need to ensure that the target server supports huge messages too. I configured following in the server for supporting huge messages.

auto.create.topics.enable=true
default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000
replica.fetch.max.bytes=104857600
message.max.bytes=104857600

RecordTooLargeException in kafka

1 Answers1