How to automatically start Kafka upon system startup in Ubuntu?

Question

Does Kafka have an official way (e.g. an init.d script) to start Kafka on system start up?

The only official method to start Kafka I have seen is:

nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &

I have tried to use a @reboot task in crontab -e however it did not start Kafka. Some people have also written custom init.d

There are also custom init.d scripts available (e.g. one, two, three) however they are all different and I am not familiar enough with init.d to understand which one, if any to implement.

How to start Kafka on system startup?

Answer 1

Here's how I configure Kafka to start automatically on Ubuntu 14.04:

sudo su
cp -R ~/kafka_2.11-0.10.0.1 /opt
ln -s /opt/kafka_2.11-0.10.0.1 /opt/kafka

Copy the following init script to /etc/init.d/kafka:

DAEMON_PATH=/opt/kafka/
PATH=$PATH:$DAEMON_PATH/bin

# See how we were called.
case "$1" in
  start)
        # Start daemon.
        echo "Starting Zookeeper";
        nohup $DAEMON_PATH/bin/zookeeper-server-start.sh -daemon /$DAEMON_PATH/config/zookeeper.properties 2> /dev/null && \
        echo "Starting Kafka";
        nohup $DAEMON_PATH/bin/kafka-server-start.sh -daemon /$DAEMON_PATH/config/server.properties 2> /dev/null
        ;;
  stop)
        # Stop daemons.
        echo "Shutting down Zookeeper";
        pid=`ps ax | grep -i 'org.apache.zookeeper.server' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Zookeeper was not Running"
        fi
        echo "Shutting down Kafka";
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Kafka was not Running"
        fi
        ;;
  restart)
        $0 stop
        sleep 2
        $0 start
        ;;
  status)
        pid=`ps ax | grep -i 'org.apache.zookeeper.server' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Zookeeper is Running as PID: $pid"
        else
          echo "Zookeeper is not Running"
        fi
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Kafka is Running as PID: $pid"
        else
          echo "Kafka is not Running"
        fi
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit 0

Make the kafka service with these commands:

chmod 755 /etc/init.d/kafka
update-rc.d kafka defaults

Now you should be able to start and stop the kafka service like this:

sudo service kafka start
sudo service kafka status
sudo service kafka stop

If you want to remove the Kafka service later, run update-rc.d -f kafka remove.

Answer 2

Download Kafka

cd /opt
sudo wget http://mirror.hosting90.cz/apache/kafka/2.5.0/kafka-2.5.0-src.tgz
sudo tar -zxvf kafka-2.5.0-src.tgz
sudo mv kafka-2.5.0-src kafka
sudo rm kafka-2.5.0-src.tgz
cd kafka
sudo ./gradlew jar -PscalaVersion=2.11.12

Install Zookeeper

sudo vi /etc/systemd/system/zookeeper.service

Edit zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=root
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Start Zookeeper

sudo systemctl enable zookeeper.service

sudo systemctl start zookeeper.service

sudo systemctl status zookeeper.service

active (running)

Install Kafka

sudo vi /etc/systemd/system/kafka.service

Edit kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=root
ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafka/kafka.log 2>&1'
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Start Kafka

sudo systemctl enable kafka.service

sudo systemctl start kafka.service

sudo systemctl status kafka.service

active (running)

Test Kafka works

create topic

sudo bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test-topic

put messages to topic

sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic  
> test message1
> test messate2
^C

read messages from topic

sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test-topic
test message1
test messate2
^C

Answer 3

One easy approach is to use systemd. You should note that at the startup the environment variables like JAVA_HOME are not loaded yet, so we should introduce them to the system. One good solution is to create a file named profile and add all the necessary variable to that:

# /home/kafka/profile
JAVA_HOME=/opt/jdk8
KAFKA_HOME=/opt/kafka

Supposing you've installed Kafka on the path /opt/kafka, in order to Kafka run automatically after Ubuntu startup (tested on Ubuntu 16.04 and centOS7 and I guess it works on any distribution with the support of the systemd) do the following command:

sudo nano /etc/systemd/system/kafka.service  # open file to add service informations

Now add the following contents to the file

[Unit]
Description=Kafka Daemon
Wants=syslog.target

# suppose you have a service named zookeeper that it start zookeeper and we want Kafka service run after the zookeeper service
After=zookeeper.service

[Service]    
Type=forking

# the user whom you want run the Kafka start and stop command under
User=kafka    

# the file path that contains envirnment variables
EnvironmentFile=/home/kafka/profile

# the directory that the commands will run there   
WorkingDirectory=/home/kafka/ 

# Kafka server start command
ExecStart=/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

# Kafka server stop command
ExecStop=/opt/kafka/bin/kafka-server-stop.sh -daemon

TimeoutSec=30
Restart=on-failure

[Install]
WantedBy=multi-user.target

Note: As Kafka need the zookeeper to connect it at the starting time, I supposed we have a zookeeper service too and I set the Kafka service to run after zookeeper service starting.

Now after saving the kafka.service file, just run the following command to create a link of the Kafka service and it will start every time you reboot the OS:

sudo systemctl enable kafka

Now you can start the Kafka service using the command:

sudo systemctl start kafka.service

and check the status of the service:

sudo systemctl status kafka.service