0

I am new to Kafka and I am trying to send messages via publisher using spring application and kafka docker container. Kafka and zookeeper is up and running but when I start springboot application it shows:

2020-06-01 18:19:05.738  WARN 4824 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.

2020-06-01 18:19:12.203  WARN 4824 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker 127.0.0.1:9092 (id: -1 rack: null) disconnected

Here you can see both containers running:

4f74ceac73ba   wurstmeister/kafka:2.12-2.5.0   "start-kafka.sh"  0.0.0.0:9092->9092/tcp kafka 

fc958792a4a0  zookeeper:3.6.1 "/docker-entrypoint.…" 2888/tcp, 3888/tcp, 0.0.0.0:2181 >2181/tcp, 8080/tcp   zookeeper

Here you can see zookeeper logs:

2020-06-01 13:17:39,518 [myid:1] - INFO  [main:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED
2020-06-01 13:17:39,519 [myid:1] - INFO  [main:FileTxnSnapLog@470] - Snapshotting: 0x0 to /data/version-2/snapshot.0
2020-06-01 13:17:39,536 [myid:1] - INFO  [main:ZKDatabase@289] - Snapshot loaded in 45 ms, highest zxid is 0x0, digest is 1371985504
2020-06-01 13:17:39,543 [myid:1] - INFO  [main:FileTxnSnapLog@470] - Snapshotting: 0x0 to /data/version-2/snapshot.0
2020-06-01 13:17:39,544 [myid:1] - INFO  [main:ZooKeeperServer@519] - Snapshot taken in 2 ms
2020-06-01 13:17:39,571 [myid:1] - INFO  [main:RequestThrottler@74] - zookeeper.request_throttler.shutdownTimeout = 10000
2020-06-01 13:17:39,671 [myid:1] - INFO  [main:ContainerManager@83] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0
2020-06-01 13:17:39,674 [myid:1] - INFO  [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.
2020-06-01 13:17:40,943 [myid:1] - INFO  [SyncThread:0:FileTxnLog@284] - Creating new log file: log.1

and docker-compose file to bootstrap containers:

version: "3.7"

networks:
  kafka-net:
    name: kafka-net
    driver: bridge

services:
  zookeeper:
    image: zookeeper:3.6.1
    container_name: zookeeper
    restart: always
    networks:
      - kafka-net
    ports:
      - "2181:2181"
    volumes:
      - c:/kafka/docker-data/zookeeper:/bitnami/zookeeper

  kafka:
    image: wurstmeister/kafka:2.12-2.5.0
    container_name: kafka
    restart: always
    networks:
      - kafka-net
    ports:
      - "9092:9092"
    volumes:
      - c:/kafka/docker-data/kafka:/bitnami/kafka
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER_INTERNAL:PLAINTEXT,DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_LISTENERS: DOCKER_INTERNAL://:29092,DOCKER_EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: DOCKER_INTERNAL://kafka:29092,DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper

There is no additional configuraion in application-properties. Any help would be appreciated, I am pretty much stuck with it :(

Kasia Michalec
  • 1
  • 1
  • 2
  • Are you able to use kafka command line producer/consumer to connect to this kafka instance ? – Amar Dev Jun 01 '20 at 16:32
  • I am able to open kafka container bash and successfully create topics. I don't know if that was your question. I am novice at this area – Kasia Michalec Jun 01 '20 at 16:37
  • If you have kafka on you local machine, you should use the shell or the bat script which is inbuilt kafka to connect to you kafka running in the container. If you are able to do so, it means you local machine can connect to you kafka running in the container. You can have a look here, https://kafka.apache.org/quickstart – Amar Dev Jun 01 '20 at 16:41
  • I don't have kafka locally installed but I am able to access kafka container and create new topics without problem – Kasia Michalec Jun 01 '20 at 16:51
  • From within the bash you will be able to do it without an issue. What I am asking you to do will verify whether you can connect from your local machine to the kafka broker. If yes, then you can debug the spring boot application. If no, then spring application might no be the actual issue. I hope this clarifies. – Amar Dev Jun 01 '20 at 17:02
  • I think that spring points to wrong port but I've tried already set locahost:9092, 0.0.0.0:9092, 127.0.0.1:9092 using property: spring.kafka.producer.bootstrap-servers – Kasia Michalec Jun 01 '20 at 17:04
  • what are the kafka properties you pass to your spring app? Also how are you connecting to kafka from your app? can you post the spring code? – robert.baboi Jun 01 '20 at 17:58
  • 1
    also, in my test code I have something like this for the server config listeners=PLAINTEXT://:9092 and reading through wurtmeisters doc, in the Advertised port section, he also mentions something similar, maybe add PLAINTEXT://:9092 to KAFKA_ADVERTISED_LISTENERS (try also with PLAINTEXT//:9092, note the missing colon) – robert.baboi Jun 01 '20 at 18:13
  • PLAINTEXT://:9092 doesn't help. Im not passing any properties inside spring application. I am only using kafkaTemplate to send message – Kasia Michalec Jun 01 '20 at 19:58
  • 'Im not passing any properties inside spring application" ... Well, you need to at least set `bootstrap.servers` **somewhere**. And your problem is the advertised listeners, not the other ones – OneCricketeer Jun 09 '20 at 03:53

0 Answers0