0

this is example how to create new 10 topic partitions with name - test_test

kafka-topics.sh --create --zookeeper zookeeper01:2181 --replication-factor 3 --partitions 10 --topic test_test

Created topic "test_test".


[root@kafka01 kafka-data]# \ls -ltr | grep  test_test
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-8
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-5
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-2
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-0
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-7
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-4
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-1
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-9
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-6
drwxr-xr-x 2 kafka hadoop  4096 Mar 22 16:53 test_test-3

now we want to add additional 10 partitions to the topic name - test_test

how to add additional partitions to the existing 10 partitions ?

jessica
  • 2,426
  • 24
  • 66

2 Answers2

2

You can run this command: 

./bin/kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic test_test --partitions 20

By the way there are two things to consider about changing partitions:

  • Decreasing the number of partitions is not allowed
  • If you add more partitions to a topic, key based ordering of the messages cannot be guaranteed

Note: If your Kafka version is older than 2.2 you must use --zookeeper parameter instead of --bootstrap-server

H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
  • I run the cli that you sent , and I get "WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected" , I little worry about the re-create the existing first 10 partitions , do you think they can be damaged ? – jessica Mar 22 '20 at 17:34
  • why - ./bin/kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic test_test --partitions 10 , not added the new partitions ? – jessica Mar 22 '20 at 17:36
  • it should be --partitions 20 not 10 – H.Ç.T Mar 22 '20 at 17:37
  • yes , but what we are worry - is about the re-created of the first 10 partitions – jessica Mar 22 '20 at 17:38
  • do you think its safe to re-create the first 10 ? and the additional 10 by the cli - /bin/kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic test_test --partitions 20 – jessica Mar 22 '20 at 17:40
  • @jessica there is no re-create. Just 10 more partitions are added. Only problem is about message ordering if you are using key based partitions and message order is important to you. – H.Ç.T Mar 22 '20 at 17:42
  • so do you mean no option to defined 10 partitions to the cli? , because they are already exists? – jessica Mar 22 '20 at 17:44
  • @jessica absolutely. If you are worried about that you can check creation time of partitions in data folder. It's absolutely safe (except message ordering) that's why it is avoided to decrease number of partitions. (to avoid data loss) – H.Ç.T Mar 22 '20 at 17:47
  • ok in that case its really very starnge that kafka except , this situation , because its very confused , I will be more happy if they add a new flag for additional partitions – jessica Mar 22 '20 at 17:49
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/210120/discussion-between-jessica-and-h-c-t). – jessica Mar 22 '20 at 17:59
1

Moreover, you should take into consideration that adding partitions triggers a rebalance which makes all of your this topic's consumers unavailable for a period of time.

rebalance is the process of re-assigning partitions to consumers, it happens when new partitions are added, new consumer is added or a consumer is leaving (may happen due to exception, network problems or initiated exit).
In order to preserve reading consistency, during a rebalance the consumer group entirely stops receiving messages until the new partition assignment is taking place.

This relatively short answer explains rebalance very well.

Ofek Hod
  • 3,544
  • 2
  • 15
  • 26