0

I read the questions here and here and this article. But can not make my brokers work together. I have 3 brokers configured with these listeners:

listeners=B1://:9092
advertised.listeners=B1://b1.kafka.myhost.com:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,B1:PLAINTEXT
inter.broker.listener.name=B1

and

listeners=B2://:9092
advertised.listeners=B2://b2.kafka.myhost.com:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,B2:PLAINTEXT
inter.broker.listener.name=B2

and a similar 3rd broker. The brokers start up fine and update their states in zookeeper as:

{"listener_security_protocol_map":{"B2":"PLAINTEXT"},"endpoints":["B2://b2.kafka.myhost.net:9092"],"jmx_port":-1,"features":{},"host":"b2.kafka.ramzinex.net","timestamp":"1679755625094","port":9092,"version":5}
{"listener_security_protocol_map":{"B1":"PLAINTEXT"},"endpoints":["B1://b1.kafka.myhost.net:9092"],"jmx_port":-1,"features":{},"host":"b1.kafka.ramzinex.net","timestamp":"1679755473055","port":9092,"version":5}
{"listener_security_protocol_map":{"B3":"PLAINTEXT"},"endpoints":["B3://b3.kafka.myhost.net:9092"],"jmx_port":-1,"features":{},"host":"b3.kafka.ramzinex.net","timestamp":"1679756711179","port":9092,"version":5}

but when i try to add a topic with:

./bin/kafka-topics.sh --bootstrap-server b1.kafka.myhost.net:9092 --create --replication-factor 3 --partitions 1 --topic test5

i get the following error:

Error while executing topic command : Replication factor: 3 larger than available brokers: 1.
[2023-03-25 15:06:23,417] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
 (kafka.admin.TopicCommand$)

I planned to use two listeners per broker for internal and external networks but can not make it work on internal network. running the same command on the b2 and b3 fails with time out. And it seems that 2,3 brokers have problem listening. But this nodes also have open sockets listening on the 0.0.0.0:9092 which are accessible from other nodes.

mhk
  • 386
  • 1
  • 5
  • 18

1 Answers1

1

You shouldn't create unique listeners per broker. No "B1", "B2", etc. Just call it "INTERNAL" and "EXTERNAL" (or just keep it called PLAINTEXT), for example, and keep it the same value on every broker

Then use this so that they all communicate on the same listener name and form a cluster

inter.broker.listener.name=INTERNAL
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks for your response. This solved the problem with one broker. How can i configure two listeners per broker one listening on localhost only (a plaintext listener for configuration purposes and an ssl listener for clients)? Am i limited to firewall configuration for limiting access to plaintext listener? – mhk Mar 25 '23 at 19:00
  • 1
    You don't need a localhost listener. Referencing the LAN ip from the same machine will always work. But the same answer applies, just keep the same "protocol" on every machine, don't give unique names – OneCricketeer Mar 25 '23 at 22:42