1

I'm running Kafka connect in standalone mode and I've started getting org.apache.kafka.common.errors.RecordTooLargeException with the message:

The message is 1259203 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

I can't seem to figure out how to increase this property and get it to stick. I've found a number of solutions via some Googling but nothing seems to work:

  • Setting it in connect-standalone.properties: producer.max.request.size=16777216
  • Setting it in producer.properties: max.request.size=16777216

After doing all of this I do:

confluent local destroy

confluent local start

confluent local log connect

And I always see this in the logs:

ost:9092 (org.apache.kafka.connect.util.TopicAdmin:230)
[2020-09-18 13:36:32,744] INFO ProducerConfig values: 
        acks = all
        batch.size = 16384
        bootstrap.servers = [localhost:9092]
        buffer.memory = 33554432
        client.dns.lookup = default
        client.id = 
        compression.type = none
        connections.max.idle.ms = 540000
        delivery.timeout.ms = 2147483647
        enable.idempotence = false
        interceptor.classes = []
        key.serializer = class org.apache.kafka.common.serialization.StringSerializer
        linger.ms = 0
        max.block.ms = 60000
        max.in.flight.requests.per.connection = 1
        max.request.size = 1048576

Notice: max.request.size = 1048576

What am I missing here?

EDIT: One thing I notice is when I start confluent I see this:

Using CONFLUENT_CURRENT: /var/folders/nh/12t_10dn2nn5tq4yy25mcyp0p04b98/T/confluent.lfzm3fZY

If I go there and go under the connect folder, and look at connect.properties. My requests.max.size is not in there at all. Should it be? How does this get populated?

It really seems like running confluent local start is just not picking up connect-standalone,properties. Is there a way to point it to that?

user1513171
  • 1,912
  • 6
  • 32
  • 54
  • Does this answer your question? [How can I send large messages with Kafka (over 15MB)?](https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb) – Shrey Jakhmola Sep 18 '20 at 17:49
  • I don't believe so since I'm using Kafka Connect which they make no mention of – user1513171 Sep 18 '20 at 18:02
  • So, `producer.properties` is not used by Connect. Refer https://docs.confluent.io/current/connect/references/allconfigs.html#override-the-worker-configuration And `confluent local` uses `connect-distributed` – OneCricketeer Sep 18 '20 at 20:13
  • @OneCricketeer, yeah that one was kind of just an attempt I didn't think would work. The connet-standalone is the issue – user1513171 Sep 18 '20 at 20:19
  • 1
    As mentioned, that file isn't used unless you use `connect-standalone.sh` script directly – OneCricketeer Sep 18 '20 at 20:20

0 Answers0