0

I am trying to run zookeeper using the Java API, so that I am able to start a topic with multiple partitions. I believe this answer does the trick. However, when I try running this code, I get the error:

Unable to connect to zookeeper server 'localhost:2181' with timeout of 8000 ms

On this machine, I don't have a zookeeper server running. But I would like to start an instance in Java (which, I am currently able to do using the ZooKeeperServerMain() class). So perhaps I need to do the same, but combining it somehow with the first answer that contains the possibility of adding multiple partitions to the topic.

For what it's worth, I am able to start zookeeper using this answer, but this will only create one partition for a topic, which is not exactly what I want.

Essentially, I want to start Zookeeper (which I am able to do using the second link), start Kafka locally (which I am also able to do), start a producer, which produces to a single topic but three different partitions, and 3 different consumers, that will read messages, one from each partition (if I understood Kafka correctly, 3 partitions will be divided equally if 3 consumers are started). All this is done using the Java API (I am not running the Kafka scripts or anything). However, I am having difficulty creating a topic with three partitions, which is my question.

Adam
  • 2,384
  • 7
  • 29
  • 66
  • How are you creating the topic? Do you create it before running the producer/consumer? Or it is automatically created by the producer/consumer? – vahid Jul 01 '17 at 23:13
  • @vahid, if I don't use the Zookeeper class (which is what I'm currently doing), the topic is created automatically by the producer using: producer.send(new ProducerRecord(topic, key, message). However, as my initial question suggests, this will create only one partition for the topic, but I would like to create 3 partitions in this same topic. – Adam Jul 01 '17 at 23:15
  • Is it possible to run your broker with `num.partitions=3` in broker config? The default for this config is 1, and that's why you end up with just a single partition for the automatically created topic. – vahid Jul 01 '17 at 23:24
  • How/where can I specify the number of partitions in the broker config? – Adam Jul 01 '17 at 23:27
  • How are you currently running your Kafka broker? – vahid Jul 01 '17 at 23:33
  • I've specified num.partitions=3 in the broker properties, but I get the error: "org.apache.kafka.common.KafkaException: Invalid partition given with record: 1 is not in the rang [0...1)." I've specified the partition value in the producer.send(), so evidently, only 1 partition is created despite specifying 3 – Adam Jul 01 '17 at 23:34
  • It sounds like 3 partitions did not get created for the topic? Did you delete the topic and start over after you added `num.partitions=3`? – vahid Jul 02 '17 at 00:16
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/148144/discussion-between-vahid-and-adam). – vahid Jul 02 '17 at 00:18

2 Answers2

2

The broker's num.partitions configuration can be used to enforce the number of partitions for automatically created topics (created by producers or consumers). For this config to take effect, the already created topic needs to be removed first. Only topics that are automatically created (after this config is changed) will be impacted.

vahid
  • 1,118
  • 9
  • 13
2

Going forward, you should not be talking directly to zookeeper to create or manage topics in Kafka. You should use the new AdminClient API introduced in Kafka 0.11.

https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html

Hans Jespersen
  • 8,024
  • 1
  • 24
  • 31