1

I'm using a .NET client (producer and consumer) of Apache Kafka.

I'm trying to increase the size of the max message that can be sent. I read the configuration manual and some posts regarding my issue:

  1. Kafka: Sending a 15MB message
  2. https://github.com/confluentinc/kafka-rest/issues/208

I saw that I should set the property "max.request.size" for the producer. So I did the following:

   var config = new Dictionary<string, object>
        {
            { "bootstrap.servers", brokerList },
            { "max.request.size", 10 * 1024 * 1024 }
        };

        using (var producer = new Producer<Null, string>(config, null, new StringSerializer(Encoding.UTF8)))
        {
         ...

But when I run the Producer, an exception is thrown :

System.InvalidOperationException: 'No such configuration property: "max.request.size"'

The Apache Kafka documentation tells the following about this property of the producer:

The maximum size of a request in bytes. This is also effectively a cap on the maximum record size. Note that the server has its own cap on record size which may be different from this. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests.

If, for some reason the documentation isn't up-to-date, what is the right property to use ?

Thanks

Community
  • 1
  • 1
Moti Berger
  • 61
  • 8

1 Answers1

2

confluent-kafka-dotnet uses librdkafka under the hood which has a different set of configuration parameters to the java client. These are documented here:

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

I believe the parameter you are looking for is message.max.bytes

Matt Howlett
  • 481
  • 3
  • 8