23

I am planning to decide on how many nodes should be present on Kafka Cluster. I am not sure about the parameters to take into consideration. I am sure it has to be >=3 (with replication factor of 2 and failure tolerance of 1 node).

Can someone tell me what parameters should be kept in mind while deciding the cluster size and how they effect the size.

I know of following factors but don't know how it quantitatively effects the cluster size. I know how it qualitatively effect the cluster size. Is there any other parameter which effects cluster size? 1. Replication factor (cluster size >= replication factor) 2. Node failure tolerance. (cluster size >= node-failure + 1)

What should be cluster size for following scenario while consideration of all the parameters 1. There are 3 topics. 2. Each topic has messages of different size. Message size range is 10 to 500kb. Average message size being 50kb. 3. Each topic has different partitions. Partitions are 10, 100, 500 4. Retention period is 7 days 5. There are 100 million messages which gets posted every day for each topic.

Can someone please point me to relevant documentation or any other blog which may discuss this. I have google searched it but to no avail

puneet
  • 530
  • 1
  • 3
  • 8
  • I want to take call on my own. I want to know if there are any parameters on basis of which we decide the cluster size. Kafka documentation doesn't provide any information about optimal cluster size. Will add data points around it. – puneet Jan 14 '15 at 11:03

3 Answers3

21

As I understand, getting good throughput from Kafka doesn't depend only on the cluster size; there are others configurations which need to be considered as well. I will try to share as much as I can.

Kafka's throughput is supposed to be linearly scalabale with the numbers of disk you have. The new multiple data directories feature introduced in Kafka 0.8 allows Kafka's topics to have different partitions on different machines. As the partition number increases greatly, so do the chances that the leader election process will be slower, also effecting consumer rebalancing. This is something to consider, and could be a bottleneck.

Another key thing could be the disk flush rate. As Kafka always immediately writes all data to the filesystem, the more often data is flushed to disk, the more "seek-bound" Kafka will be, and the lower the throughput. Again a very low flush rate might lead to different problems, as in that case the amount of data to be flushed will be large. So providing an exact figure is not very practical and I think that is the reason you couldn't find such direct answer in the Kafka documentation.

There will be other factors too. For example the consumer's fetch size, compressions, batch size for asynchronous producers, socket buffer sizes etc.

Hardware & OS will also play a key role in this as using Kafka in a Linux based environment is advisable due to its pageCache mechanism for writing data to the disk. Read more on this here

You might also want to take a look at how OS flush behavior play a key role into consideration before you actually tune it to fit your needs. I believe it is key to understand the design philosophy, which makes it so effective in terms of throughput and fault-tolerance.

Some more resource I find useful to dig in

slm
  • 15,396
  • 12
  • 109
  • 124
user2720864
  • 8,015
  • 5
  • 48
  • 60
5

I had recently worked with kafka and these are my observations.

Each topic is divided into partitions and all the partitions of a topic are distributed across kafka brokers; first of all these help to save topics whose size is larger than the capacity of a single kafka broker and also they increase the consumer parallelism.

To increase the reliability and fault tolerance,replications of the partitions are made and they do not increase the consumer parallelism.The thumb rule is a single broker can host only a single replica per partition. Hence Number of brokers must be >= No of replicas

All partitions are spread across all the available brokers,number of partitions can be irrespective of number of brokers but number of partitions must be equal to the number of consumer threads in a consumer group(to get best throughput)

The cluster size should be decided keeping in mind the throughput you want to achieve at consumer.

Nithin
  • 328
  • 1
  • 7
  • 16
  • To achieve high consumer throughput i.e. consume messages at high rate, increase the number of partitions and fire no of threads equal to no of partitions in highlevel consumer. – Nithin Jan 15 '15 at 17:08
  • @nitin what will happen if you have like 1000 partitons and want to run consumers to consumer messages from all partitions ??? – user2720864 Jan 16 '15 at 10:56
  • 1
    Design a highlevel consumer with 1000 threads so that each thread consumes one partition. For more details see "Designing a Highlevel Consumer' in this link --> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example – Nithin Jan 16 '15 at 13:26
  • As per my research, increasing partitions impact more performance than cluster size. If you are increasing number of partitions as 1000 you should be careful with consumers. If all consumers are running from same machine it may be a bit of worry (as 1000 new threads will be spawned). If your machines are good enough than it's ok, otherwise you need to run consumers in more than one machine (with same consumer groups) – puneet Jan 18 '15 at 19:57
  • @nithin in that case consuming from all partitions will cause the consumer rebalancing much much slower greatly effecting the throughput which is the primary concern here – user2720864 Jan 18 '15 at 19:57
  • 1
    @user2720864 as mentioned by leonardo run multiple machines with same consumer group. No of partitions should be decided by keeping in mind the size of the topic and to balance it's load on brokers. Yes, more partitions increase consumer throughput but assigning a large number of partitions leads to very less data/partition and more thread overhead – Nithin Jan 19 '15 at 09:59
5

The total MB/s per broker would be:

Data/Day = (100×10^6 Messages / Day ) × 0.5MB = 5TB/Day per Topic

That gives us ~58MB/s per Broker. Assuming that the messages are equally split between partitions, for the total cluster we get: 58MB/s x 3 Topics = 178MB/s for all the cluster.

Now, for the replication, you have: 1 extra replica per topic. Therefore this becomes 58MB/sec/broker INCOMING original data + 58MB/sec/broker OUTGOING replication data + 58MB/sec/broker INCOMING replication data.

This gets about ~136MB/s per broker ingress and 58MB/s per broker egress.

The systems load will get very high and this is without taking into consideration any stream processing.

The system load could be handled by increasing the number of brokers and splitting your topics to more specific partitions. If your data are very important, then you may want a different (high) replication factor. Fault tolerance is also an important factor for deciding the replication.
For example, if you had very very important data, apart from the N active brokers (with the replicas) that are managing your partitions, you may require to add stand-by followers in different areas. If you require very low latency, then you may want to further increase your partitions (by adding additional keys). The more keys you have, the fewer messages you will have on each partition. For low latency, you may want a new cluster (with the replicas) that manages only that special topic and no additional computation is done to other topics. If a topic is not very important, then you may want to lower the replication factor of that particular topic and be more elastic to some data loss. When building a Kafka cluster, the machines supporting your infrastructure should be equally capable. That is since the partitioning is done with round-robin style, you expect that each broker is capable of handling the same load, therefore the size of your messages does not matter.

The load from stream processing will also have a direct impact. A good software to manage your kafka monitor and manage your streams is Lenses, which I personally favor a lot since it does an amazing work with processing real-time streams

lloiacono
  • 4,714
  • 2
  • 30
  • 46
ElisaLav
  • 51
  • 1
  • 1