0
docker run -it --rm --name kafka -p 9092:9092 -p 9094:9094 --link zookeeper:zookeeper --env KAFKA_LISTENERS=LISTENER_INTERNAL://:9092,LISTENER_EXTERNAL://:9094 --env KAFKA_ADVERTISED_LISTENERS=LISTENER_INTERNAL://kafka:9092,LISTENER_EXTERNAL://localhost:9094 --env KAFKA_ALLOW_PLAINTEXT_LISTENER="yes" --env KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT --env KAFKA_INTER_BROKER_LISTENER_NAME=LISTENER_INTERNAL  debezium/kafka:1.9

I am trying to start Kafka via debezium's docker image. I have KafkaConsumers both within the docker network and outside the docker network.

If I start Kafka with

--env KAFKA_ADVERTISED_LISTENERS=LISTENER_INTERNAL://:9092,LISTENER_EXTERNAL://localhost:9094

My external consumers work fine by connecting to port 9094.

Now my docker internal consumers don't work if LISTENER_INTERNAL is set to :9092. All examples I have seen use docker compose and they do LISTENER_INTERNAL://kafka:9092

If I try that (original snippet), then I get the following errors and they refuse to start.

java.net.UnknownHostException: kafka
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:988)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:64)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:291)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:245)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

How do I tell docker to use the IP address of the container or the name of the container as it starts and bind it to LISTENER_INTERNAL?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Anand S
  • 87
  • 1
  • 10

1 Answers1

1

You don't need an IP address of the container.

You've either not used --link (which is deprecated) or not used --network on a shared bridge.

If you want to make Kafka listen on all interfaces, you'd set listeners=<PROTOCOL>://0.0.0.0:<port>, but by default, it should bind to its hostname, and that won't solve DNS errors for external clients

Related post Connect to Kafka running in Docker

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Hi @OneCricketeer, thanks. This is how I start it. ```docker run -it --rm --name kafka -p 9092:9092 -p 9094:9094 --link zookeeper:zookeeper --env KAFKA_LISTENERS=INTERNAL://:9092,OUTSIDE://:9094 --env KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,OUTSIDE://localhost:9094 --env KAFKA_ALLOW_PLAINTEXT_LISTENER="yes" --env KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT --env KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL debezium/kafka:1.9 ``` – Anand S Aug 28 '22 at 16:04
  • See that I am trying to start `kafka` container so I can't link it to itself yet. All the examples I see use docker-compose, which supports it. You can specify the name of the container as an advertised listener. – Anand S Aug 28 '22 at 16:05
  • 1
    Ah I suppose it is the `--network`. Let me check. – Anand S Aug 28 '22 at 16:10
  • 1
    That was it. I had to create a docker network and stand up all the containers within that network. Thanks. – Anand S Aug 28 '22 at 17:48
  • We wrote a layer above docker API to stand up these clusters so I have to use that to automate different types of clusters. Thanks for the help. – Anand S Aug 28 '22 at 22:16