1

Knative configuration as below:

  • Source: AWS SQS source
  • Broker: Kafka broker
  • Trigger: A trigger with sink is a Knative service
  • Knative service: A knative service with containerConcurrency = 1 (to make sure 1 request/pod at a time)

Send 1000 messages to SQS queue, Knative service only creates about 100-150 pods to process all requests.
I checked and found that Kafka broker only sends about 100-180 events to Knative Service at once.
Thus, Knative service only creates about 1xx pods to handle these events.
The remaining events will be failed to send to Knative service and broker will retry sending.
Is there any configuration in Kafka broker to increase number of events that will be sent to Knative service at once?

FYI:

  • With MTchannelbased broker (in-memory channel), it can send 400-1000 events to Knative service at once.
  • I use Knative v1.7.1 and Knative Kafka broker v1.7.0
xuanhai266
  • 423
  • 3
  • 13

3 Answers3

1

Isn't because you have containerConcurrency=1?

Broker receives events from SQS and commits the offset of received events xx seconds (auto.commit.interval.ms).

Now, with containerConcurrency means that every event which is send to the kservice can be consumed one per whole consumer. If you will increase this number you will also increase the numer of events possible to be send.

If you care about the order per-partition you can play and use kafka.eventing.knative.dev/delivery.order setting which will make per-partition sequential execution of events, one event in the partition is successfully processed, a next one will be send. Other partitions are processed async.

Piotr
  • 21
  • 1
1

I confirmed that we can use 2 parameters:

  • max.poll.records
  • maxPoolSize

https://github.com/knative-sandbox/eventing-kafka-broker/issues/2597#issuecomment-1243665444

xuanhai266
  • 423
  • 3
  • 13
1

It looks like you're hitting concurrency limits with your Kafka topic and delivery here; in particular, Kafka delivery parallelism (how many events can be delivered at once) is probably bounded by partitions * event-batch-size. It looks like the number of partitions is controlled by the kafka-broker-config ConfigMap and the latter is controlled by max.poll.records in the config-kafka-broker-data-plane ConfigMap.

Another limiting factor of the Kafka implementation is the number of outgoing HTTP requests from a single dispatcher, which appears to be controlled by the maxPoolSize parameter of the data plane.

You didn't see this behavior from the InMemory channel, because it doesn't perform ordered delivery from partitions, and simply attempts to deliver all messages in parallel without tracking order or blocking newer messages on older ones.

E. Anderson
  • 3,405
  • 1
  • 16
  • 19
  • Yes, you're correct. I changed the values of 2 parameters: max.poll.records and maxPoolSize. And it works well. – xuanhai266 Sep 14 '22 at 09:44
  • I wanted to explain for future readers _why_ you need to tune those parameters. I used your answer to figure out the parameter names, since the docs are sparse. – E. Anderson Sep 15 '22 at 01:30