1

When attempting to Dockerize my application, the images and container properly get created. However, when attempting to utilize Docker Landoop Kafka Cluster, my microservice applications (EmployeeDataWriter and EmployeeDataReader) can not connect to the broker.

Please find my docker-compose.yml, kafka producer configs, kafka consumer configs listed below:

version: '3'

services:

  # Eureka Server
  EurekaServer:
    image: eureka_server
    ports:
      - 8761:8761
    restart:
      always

  # Employee Data Writer
  EmployeeDataWriter:
    image: employee_data_writer
    depends_on:
      - EurekaServer
      - kafka-cluster
    environment:
      - eureka.client.service-url.defaultZone=http://EurekaServer:8761/eureka
      - spring.datasource.url=jdbc:mysql://host.docker.internal:3306/employeedata
      - spring.datasource.username=root
      - spring.datasource.password=[PLACEHOLDER]
      - spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
      - spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
      - spring.jpa.hibernate.ddl-auto=none
      - spring.jpa.show-sql=true
    links:
      - EurekaServer
    ports:
      - 8070:8070

  # Employee Data Reader
  EmployeeDataReader:
    image: employee_data_reader
    depends_on:
      - EurekaServer
      - kafka-cluster
    environment:
      - eureka.client.service-url.defaultZone=http://EurekaServer:8761/eureka
      - spring.data.mongodb.uri=mongodb://host.docker.internal:27017/EMPLOYEEDATA
      - spring.data.mongodb.database=EMPLOYEEDATA
    links:
      - EurekaServer
    ports:
      - 8072:8072

  # Kafka Cluster.
  kafka-cluster:
    image: landoop/fast-data-dev:cp3.3.0
    environment:
      ADV_HOST: 127.0.0.1         # Change to 192.168.99.100 if using Docker Toolbox
      RUNTESTS: 0                 # Disable Running tests so the cluster starts faster
      FORWARDLOGS: 0              # Disable running 5 file source connectors that bring application logs into Kafka topics
      SAMPLEDATA: 0               # Do not create sea_vessel_position_reports, nyc_yellow_taxi_trip_data, reddit_posts topics with sample Avro records.
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect ports
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker

properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
properties.setProperty("schema.registry.url", "http://127.0.0.1:8081");
properties.setProperty(ProducerConfig.ACKS_CONFIG, "all");


properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
    StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
    KafkaAvroDeserializer.class.getName());
properties.setProperty("schema.registry.url", "http://127.0.0.1:8081");
properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "EmployeeDataConsumer");
properties.setProperty("specific.avro.reader", "true");
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

The error message in the docker log for the microservice applications are

2022-04-29 21:16:29.919 WARN 1 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

2022-04-29 21:16:30.073 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

2022-04-29 21:03:53.448 WARN 1 --- [ main] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-EmployeeDataConsumer-1, groupId=EmployeeDataConsumer] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

Omar Kazimi
  • 11
  • 1
  • 2
    My suspicion is that the `ADV_HOST` is wrong. The java-application tries to connect to `127.0.0.1`, which is the container running the java-applicaction (where no kafka service is running). It should be the name of the kafka-container (i.e. `kafka-cluster`). Likewise, the java-application should be configured to connect to the bootstrap server (i.e. `kafka-cluster`), not to `localhost`. – Turing85 Apr 29 '22 at 21:29
  • Similarly, the Schema Registry is wrong as well as it should point at the Landoop container. Also note that it's running a rather old version of the Confluent components – OneCricketeer May 01 '22 at 04:55
  • @Turing85 Your recommended solution worked, your help is much appreciated. Thank you!!!! – Omar Kazimi May 02 '22 at 14:33

0 Answers0