Increasing number of brokers in kafka using KAFKA_ADVERTISED_HOST_NAME

Question

I'm new to Kafka and I'm trying to run a Kafka service on my local machine and use it to transfer some data from one .NET project to another. I'm using docker-compose.yml file to create two docker containers for zookeeper and Kafka from wurstmeister. In Kafka definition in environment variables there is KAFKA_ADVERTISED_HOST_NAME which I set to 127.0.0.1. In docker hub of wurstmeister/kafka says I quote

"modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host IP if you want to run multiple brokers.)".

When I set my topic to have more than 1 replica and more than 1 partition I get this message

"Error while executing topic command : Replication factor: 2 larger than available brokers: 1.".

What is the right IP address to define in KAFKA_ADVERTISED_HOST_NAME which will allow me to get more than 1 broker?

version: '2'
services:
 zookeeper:
  image: wurstmeister/zookeeper
  ports:
   - "2181:2181"
 kafka:
  image: wurstmeister/kafka
  ports:
   - "9092:9092"
  environment:
   KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
   KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock

OneCricketeer · Accepted Answer · 2021-12-09T03:15:28.923

Note - Running multiple brokers on one machine does not provide true fault tolerance.

use it to transfer some data from one .NET project to another

You only need one broker for this.

First, suggest reading https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

KAFKA_ADVERTISED_HOST_NAME is deprecated. Don't use it.

It has been replaced by KAFKA_ADVERTISED_LISTENERS, which can both contain ${DOCKER_HOST_IP:-127.0.0.1} (or your host IP from ifconfig) since this is the IP your client will use (from the host).
Since brokers are clients of themselves, you also need to advertise the container names. And, if you want to containerize your client, those addresses would also be used by your app.

Beyond Kafka networking configs, KAFKA_BROKER_ID needs to be different between each, and your error is suggesting you need to override all the other replication factor broker configs.

All in all.

version: '3'
services:
 zookeeper:
  image: wurstmeister/zookeeper
 kafka-1:
  image: wurstmeister/kafka
  depends_on: [zookeeper]
  ports:
   - "9092:9092"
  environment:
   KAFKA_BROKER_ID: 1
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9092,PLAINTEXT_INTERNAL://kafka-1:29092
   KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://:29092
   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
   KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
   KAFKA_DEFAULT_REPLICATION_FACTOR: 2
   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
 kafka-2:
  image: wurstmeister/kafka
  ports:
   - "9093:9093"
  environment:
   KAFKA_BROKER_ID: 2  # unique
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9093,PLAINTEXT_INTERNAL://kafka-2:29093
   KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,PLAINTEXT_INTERNAL://:29093
   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
   KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
   KAFKA_DEFAULT_REPLICATION_FACTOR: 2
   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
   KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
  depends_on:  # ensure this joins the other
   - zookeeper
   - kafka-1

I'm trying to run a Kafka service on my local machine

Set bootstrap.servers="localhost:9092,localhost:9093"

Further reading - Connect to Kafka running in Docker

Example usage

Note that topic creation doesn't seem to automatically work. You should ideally be using your application to actually create / check topic existence.

$ kcat -b localhost:9093 -L
Metadata for all topics (from broker -1: localhost:9093/bootstrap):
 2 brokers:
  broker 2 at 127.0.0.1:9093
  broker 1 at 127.0.0.1:9092 (controller)
 0 topics:

producing some data... and consuming it

$ kcat -b localhost:9093 -t test -o earilest
% Auto-selecting Consumer mode (use -P or -C to override)
hello
world
sample
data
of
no
importance

Thank you so much it works for me for 2 brokers, where can i find the full list of environment variables that are required to run multiple brokers and add some extra stuff inside my docker containers — Nikola Nikolic, Dec 10 '21 at 13:32
@Nikola There's not really a source of truth for such information outside the Kafka documentation. If you want to simply run a Kafka cluster, a cloud hosted solution would be best. Or if you want to use Kubernetes, then use Strimzi operator. Compose doesn't provide the same feature set as either option. Adding "extra stuff inside" will require making your own Docker images — OneCricketeer, Dec 12 '21 at 04:27

score 0 · Answer 2 · edited Dec 09 '21 at 03:20

0

it should be ip address of your machine. Linux: use ifconfig command and your IP will be inet <IP_ADDRESS> in windows you will get this by using ipconfig

edited Dec 09 '21 at 03:20

OneCricketeer

179,855
19
132
245

answered Dec 08 '21 at 16:35

vaibhav shanker

173
2
10

This answers the question, but it won't get the containers working together – OneCricketeer Dec 09 '21 at 03:20

Increasing number of brokers in kafka using KAFKA_ADVERTISED_HOST_NAME

2 Answers2

Example usage