Start Zookeeper and Kafka through java code

Question

Yesterday I asked this question and I'm trying to follow some other answers I've found but they're leading me nowhere. I really can't understand how to properly set up Zookeeper and then Kafka server from code. What I did so far was this:


Properties prop = new Properties();

prop.setProperty("dataDir","C:\\kafka_2.12-2.2.0\\config\\zookeeper.properties");
prop.setProperty("bootstrap.servers", "localhost:2181");

QuorumPeerConfig quorumConfiguration = new QuorumPeerConfig();

try {
     quorumConfiguration.parseProperties(prop);
} catch(Exception e) {
   throw new RuntimeException(e);
}
            
ZooKeeperServerMain zookeeper = new ZooKeeperServerMain();
            
final ServerConfig configuration = new ServerConfig();

configuration.readFrom(quorumConfiguration);
            
            new Thread() {
                public void run() {
                    try {
                        zookeeper.runFromConfig(configuration);
                    } catch (IOException e) {
                        
                    }
                }
            }.start();
            
                    
             Properties props = new Properties();
             props.put("bootstrap.servers", "localhost:9092");
             props.put("acks", "all");
             props.put("retries", 0);
             props.put("batch.size", 16384);
             props.put("linger.ms", 1);
             props.put("buffer.memory", 33554432);
             props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
             props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    
             KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
             for (int i = 0; i < 100; i++)
                 producer.send(new ProducerRecord<String, String>("info", Integer.toString(i), Integer.toString(i)));

producer.close();

I couldn't find anything more than this.

@howie I want to start zookeeper and kafka directly from java code without using cmd line — Usr, Apr 03 '19 at 13:56
Basically yes, I'm working with localhost and until now I've always started them from command line — Usr, Apr 03 '19 at 14:02
There is zero sense in what you're trying to do. Start your Zookeeper and Kafka separately as is intended. — daniu, Apr 03 '19 at 14:14
@daniu what do you mean? I'm trying to start them separately from code but this is all I got until now — Usr, Apr 03 '19 at 14:15
No, you're trying to start them within threads in your application. Start them in separate processes, like you would have when you started them from the CL. — daniu, Apr 03 '19 at 14:19
I've tried that by using ProcessBuilder, but it doesn't start the processes and I don't know how to specify that Kafka server process has to start after the other one is correctly started — Usr, Apr 03 '19 at 14:23

score 1 · Answer 1 · answered Apr 03 '19 at 20:39

1

Using the *-server-start commands would be highly recommended... They setup the classpath correctly.

Otherwise, looks like you are trying to write just simple tests, and libraries exist already for Embedded Kafka. e.g. https://github.com/embeddedkafka/embedded-kafka or using Docker

answered Apr 03 '19 at 20:39

OneCricketeer

179,855
19
132
245

Hi, not I'm not writing tests, I simply want to start them directly from java when I start the program... anyway thank you for the hint – Usr Apr 04 '19 at 08:08
You can still use the same concepts if you look into the source code of those classes. For Kafka, the main class is `KafkaServerStartable`... And Junit is not required, just a different way to have a main method. You can see another example here https://gist.github.com/fjavieralba/7930018 ... In your question, you only started Zookeeper, not Kafka, so the KafkaProducer wouldn't be working. – OneCricketeer Apr 04 '19 at 16:29

Start Zookeeper and Kafka through java code

1 Answers1