0

I have a docker-compose file that creates 3 Kafka nodes and 1 topic:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092"
    environment:
      HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2" # Normal instances
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: ${IPADDRESS}
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE  
      KAFKA_CREATE_TOPICS: "Upload_Kafka_Topic:1:3" # 1 partition and 3 replicas
    volumes:
      - /docker-volumes/run/docker.sock:/var/run/docker.sock

Now I have an application.yml file as part of my Spring Project that gets the Broker IP Adresses injected:

spring:
  kafka:
    consumer:
      group-id: upload-group
      auto-offset-reset: earliest
    bootstrap-servers: ${BROKERS_IP_ADDRESSES}

when I inject BROKERS_IP_ADDRESSES with for instance localhost:9092 or anything alike I get a connection error saying:

[Consumer clientId=json-0, groupId=upload-group] Connection to node -1 (/localhost:9092) could not be established. Broker may not be available

But when I use a custom script to insert each individual broker (or even just manually a single one!) into the BROKERS_IP_ADDRESSES, I also get an error, but a java.net.UnknownHostException with exactly 3 different hashes something like: 8eedde00f315 (matching the count of my broker addresses)

I am assuming my second approach works and is delegated to Kafka, which uses a different internal handling of the brokers, but exposes these hashes to my application, which fails to make the correct look up of brokers in return.

Is there some additional configuration or a configuration of my Kafka environment that would resolve this look up error?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Mr. M
  • 37
  • 4
  • 1) Run the spring code in a container or you need to connect to port 9094 since that's the "outside listener" (and you need to forward it in the compose file 2) You don't need to build that wurstmeister image 3) That compose file only runs one broker, but running multiple on one machine doesn't improve or change how the protocol works – OneCricketeer Nov 22 '20 at 17:36
  • 1) If i forward the port I get: The "kafka" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash. 2) I am building the Dockerfile in the same folder – Mr. M Nov 22 '20 at 18:27
  • 1) Read the repo wiki how you're supposed to build a cluster, or don't use one machine for it 2) `image: wurstmeister/kafka` works as well – OneCricketeer Nov 22 '20 at 19:23

0 Answers0