4

In designing a streaming processing pipeline what cost might be incurred if I were to have many topics which would have at least one partition but potentially no data going into it?

As an example, with one consumer and I could choose to have one "mega topic" which contains all of the data and many partitions or I could choose to split that data (by tenant, account, or user etc.) into many topics with, by default, a single partition. My worry about the second case is that there would be many topics/partitions which would see no data. So, is this unused partition costing anything or is there no cost that is incurred by an unused topic.

cwadding
  • 870
  • 11
  • 16

2 Answers2

9

First of all, there is no difference between one fat topic and lots of partitions and more than one topic that contains a few partitions. Topic is just for logical distinction between events. Kafka only cares about number of partitions.

Secondly, having lots of partitions can lead some problems:

  • Too many open files:

Each partition maps to a directory in the file system in the broker. Within that log directory, there will be two files (one for the index and another for the actual data) per log segment.

  • More partitions requires more memory both in broker and consumer sides:

Brokers allocate a buffer the size of replica.fetch.max.bytes for each partition they replicate. If replica.fetch.max.bytes is set to 1 MiB, and you have 1000 partitions, about 1 GiB of RAM is required.

  • More Partitions may increase unavailability:

If a broker which is controller is failed, then zookeeper elect another broker as controller. At that point newly elected broker should read metadata for every partition from Zookeeper during initialization.

For example, if there are 10,000 partitions in the Kafka cluster and initializing the metadata from ZooKeeper takes 2 ms per partition, this can add 20 more seconds to the unavailability window.

You may get more information from these links:
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ https://docs.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html

H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
  • 1
    Just to update, (and if you are using confluent cloud) as of 1st July 2020 Confluent charges per 'Partition-hour' as well. Which can add up to quite a bit. – Joe M Jul 21 '20 at 05:44
2

Assuming the mentioned topics are not compacted, there is the initial overhead of retaining any initially produced data, but after which, an empty topic is just

  1. metadata in zookeeper
  2. metadata in any consumer group coordinator, and wasted processing by any active consumer threads
  3. empty directories on disk

For the first two, having lots of topics may increase request latency, causing an unhealthy cluster.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245