4

I have a Kafka Streams application that consumes from topic 'A' with 10 partitions with around 10k messages per second. I am confused about what will be better for my application.

To run multiple Kafka Streams application instances with same consumer group.
  OR 
To run single kafka streams application with more num.stream.threads
Nandish Kotadia
  • 441
  • 1
  • 6
  • 21

1 Answers1

3

As mentioned in the confluent blog

The maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by maximum number of partitions of the input topic(s) the application is reading from. For example, if your input topic has 5 partitions, then you can run up to 5 applications instances.

So there is no difference when you run 10 application instance or a single streams application with 10 thread in processing of messages. Except when you run 10 application instance you can run it on different JVMs spread across different machines which can help in some throughput improvement.

Also see this

asolanki
  • 1,333
  • 11
  • 18