How to proper set up docker compose of kafka producer?

Question

I am converting test project to docker

I have working containers: Mysql,Application,Kafka.

Im having error in zookeeper or maybe wrong set up in docker-compose.yaml

//docker-compose.yaml

version: '3.8'

networks:
    product-net:
      driver: bridge 
      
services: 
   
  productmicroservice:
    image: productmicroservice:latest
    container_name: productmicroservice
    depends_on:
      - product-mysqldb
      - kafka
    restart: always
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - "9001:8091"
    
    environment:
    
      MYSQL_HOST: product-mysqldb
      MYSQL_USER: root
      MYSQL_PASSWORD: root
    links:
      - kafka:localhost
    networks:
      - product-net
      

  product-mysqldb:
    image: mysql:8.0.28
    restart: unless-stopped
    container_name: product-mysqldb
    ports: 
      - "3307:3306"
    cap_add:
      - SYS_NICE
    environment:
      MYSQL_DATABASE: dbpoc
      MYSQL_ROOT_PASSWORD: root
    
    networks:
      - product-net
      
      
  zookeeper:
    image: wurstmeister/zookeeper:latest
    container_name: zookeeper
    restart: on-failure
    ports:
      - "2181:2181"
    networks:
      - product-net
    
    
  kafka:
    image: wurstmeister/kafka:2.11-1.1.1
    restart: unless-stopped
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_MESSAGE_MAX_BYTES: 2000000
      KAFKA_CREATE_TOPICS: "producttopic:1:1"
      BROKER_ID: 1
      ADVERTISED_PORT: 9092
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - product-net
    depends_on:
      - zookeeper

//Dockerfile

FROM openjdk:8
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

//application.yaml

spring:
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://${MYSQL_HOST:localhost}:${MYSQL_PORT:3306}/dbpoc
    username: root
    password: root
  
  kafka:
    template:
      default-topic: producttopic
    producer:
      bootstrap-servers:
      - localhost:9092
      key-serializer:
        org.apache.kafka.common.serialization.StringSerializer
      value-serializer:
        org.springframework.kafka.support.serializer.JsonSerializer
    
  jpa:
    hibernate:
      naming:
        implicit-strategy: org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
        physical-strategy: org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy
    hibernate.ddl-auto: update
    generate-ddl: "false"
    show-sql: "false"
    properties:
      hibernate:
        dialect: org.hibernate.dialect.MySQL5InnoDBDialect

  mvc:
   throw-exception-if-no-handler-found: "true"
  web:
    resources:
      add-mappings: "false"  
        
  sql:
    init:
      mode: always
      continue-on-error: "true"

    
server:
  port: 8091

//Error show

kafka                | creating topics: producttopic:1:1
zookeeper            | 2022-05-03 12:21:55,223 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /172.27.0.4:34150
zookeeper            | 2022-05-03 12:21:55,225 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /172.27.0.4:34150
zookeeper            | 2022-05-03 12:21:55,229 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@694] - Established session 0x100008559dd0003 with negotiated timeout 30000 for client /172.27.0.4:34150
zookeeper            | 2022-05-03 12:21:55,377 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x100008559dd0003
zookeeper            | 2022-05-03 12:21:55,381 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /172.27.0.4:34150 which had sessionid 0x100008559dd0003



//after i try to send a data

2022-05-03 12:15:52.695  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

Do not use `kafka:localhost` as a container link. Seems like you might want to read the Compose networking documentation available from Docker after reading marked duplicate post — OneCricketeer, May 03 '22 at 13:20
Also, nothing says ERROR in the Zookeeper logs, so why are you saying there is one? — OneCricketeer, May 03 '22 at 14:03
Sorry im new , you are right its not an error. based on my research there must also changes in spring-boot project in order to communicate with docker-compose same like what i did to the mysql. For today im still studying how to it and looking for sample project in github. btw i able to run kafka and zookeeper without implementing the appilication in docker-compose but in project i need to implement it. Thank you for your effort, for now its getting clearer to me that what i did wrong, this is why my post question is confusing. — , May 04 '22 at 03:17
The duplicate post links to a fully defined and working compose file for Kafka and Zookeeper services. All you need to do is add in your mysql and spring services — OneCricketeer, May 04 '22 at 13:59

score 0 · Answer 1 · edited May 03 '22 at 13:18

0

you have to set the ADVERTISE_LISTENERS properly for the internal and external docker network, something like:

      KAFKA_ADVERTISED_LISTENERS: >- 
        LISTENER_DOCKER_INTERNAL://kafka:19092, 
        LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-localhost}:9092

you can find a working example here: https://github.com/stockgeeks/docker-compose/blob/master/one-to-run-them-all/docker-compose.yml

And an article with explanation here: https://dev.to/thegroo/one-to-run-them-all-1mg6

And some explanation on the advertise addresses in this article: https://dev.to/thegroo/running-kafka-on-kubernetes-for-local-development-with-storage-class-4oa9 in the ned the section "Connecting a Kafka client"

edited May 03 '22 at 13:18

OneCricketeer

179,855
19
132
245

answered May 03 '22 at 13:06

groo

4,213
6
45
69

That's not the only needed change. The spring app still points at localhost, for example – OneCricketeer May 03 '22 at 13:19
If you advertise the external address to localhost and run the app in your host machine(not in docker network) while exposing the port to the host that would work. I am assuming he is running his app from the host machine. i.e - Development IDE – groo May 03 '22 at 13:21
Then why is there a Dockerfile for a Java app in the question? And a `productmicroservice` service in Compose that depends on the Kafka container? – OneCricketeer May 03 '22 at 13:39
Yeah, it's a confusing question as he mentions: Implementation Working container: Mysql,Application,Kafka. Implementation Not Working container: Zookeeper So I assumed he was only running those in docker-compose and the app for dev in the IDE. – groo May 03 '22 at 13:59
The Zookeeper container is working fine. If it wasn't, then Kafka container wouldn't start, and the app would fail regardless of how Kafka or itself was configured – OneCricketeer May 04 '22 at 02:06

How to proper set up docker compose of kafka producer?

1 Answers1