0

I have been facing some problems dealing with Kafka, I'm a begginer and I'm still learning, so I would appreciate any help.

In my particular case I'm working with Kafka and docker in a compose file. Here I present my compose file:

version: "3.2"
services:
####################################
  zookeeper:
    image: 'confluentinc/cp-zookeeper:latest'
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181

###############################################################
  broker:
    image: 'confluentinc/cp-kafka:latest'
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      ALLOW_PLAINTEXT_LISTENER: "yes"
      # Exposes 9092 for external connections to the broker
      # Use kafka:29092 for connections internal on the docker network
      # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
      KAFKA_LISTENERS: "PLAINTEXT://:29092,PLAINTEXT_HOST://:9092"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      ################################################################
  kafdrop2:
    image: 'obsidiandynamics/kafdrop:latest'
    container_name: kafdrop2
    ports:
      - 9000:9000
 

    environment:
      KAFKA_BROKERCONNECT: broker:29092

Additionally, I'm using Kafdrop to see my topics and messages faster instead of using the command line.

I have created a topic called First_Topic, and I have the following code:

package com.nrich.curse.kafka.tutorial;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class demo {
    public static void main(String[] args) {
        // Create the producer properties
        String servers = "broker:29092";
        Properties properties = new Properties();
        // properties.setProperty("bootstrap.servers", "broker:9092");
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
        // Serializer helps the producer know what type of value is send
        // and how to serialize it to bytes
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());


        // Create the producer, using String in each one mean the key/value be a string
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);

        // Creating the producer record
        ProducerRecord<String, String> record =
                new ProducerRecord<String, String>("First_Topic","hello world nuevo nuevo funcionando");

        // Send Data - COnsider it is asyncronic
        producer.send(record);
        // flush the data
        producer.flush();
        // flush and close the producer
        producer.close();
    }
}

My problem is that when I set the bootstrap-server or direction using this: String servers = "broker:29092"; and I try to run the jar file using docker it fails. The jar file was compiled with maven, so what I did was to put inside the docker file the jar.

My docker file looks like this:

FROM openjdk:11
WORKDIR /
COPY kafka-course-1.0.jar ./

CMD  java -Xmx10m -Xms10m -jar kafka-course-1.0.jar

I have build it, and when I run it,I always get the following error:

[main] WARN org.apache.kafka.clients.ClientUtils - Couldn't resolve server broker:29092 from bootstrap.servers as DNS resolution failed for broker
[main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 0 ms.
[main] INFO org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed
[main] INFO org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
[main] INFO org.apache.kafka.common.metrics.Metrics - Metrics reporters closed
[main] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.producer for producer-1 unregistered
Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka producer
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:442)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:292)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:319)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:304)
    at com.nrich.curse.kafka.tutorial.demo.main(demo.java:24)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89)
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)

I have no idea why it happens, but it does not work.

What I saw is that if I run the code without docker, just running it locally it works, but it only works when I change the bootstrap-server like this: String servers = "localhost:9092";. In fact, I have tried it using the new jar file inside the docker (changing the bootstrap-server and compiling it again) but even thought it does not work.

As I mentioned, if I run it locally, it works using the "localhost:9092", and the producer gets the message inside the topic as you can see in the picture below: Events inside a topic

If appreciate any solution or advice. Thank you!

Alex_26r27
  • 3
  • 3
  • 1
    Are you running the docker image as part of the Docker Compose, or separately? – Robin Moffatt Mar 02 '22 at 09:12
  • I have tried both of them, runing it with the Docker Compose as well as separately. Do you need it? I can provide the code as well. – Alex_26r27 Mar 02 '22 at 16:34
  • Your hint is `DNS resolution failed` ... This would not happen if you ran the image in the **same Compose file**. Or if you use `docker run --network` option. FWIW, dont hardcode your bootstrap servers. Get the information from a config file that is loaded through an environment variable... One more thing: With Java 11, you should be using Xmx/Xms inside containers. You should also look into the Jib Maven plugin rather than write your own Dockerfile – OneCricketeer Mar 02 '22 at 20:08
  • Thank you, but could you share (if it's possible) the resources or where I can see that? I'm still learning docker, so I have no idea at all about it, but I will do my research. – Alex_26r27 Mar 03 '22 at 02:01
  • If you want to run it in Docker then you can either deploy it within the same Docker Compose, in which case use `broker:29092`. If you want to run it in Docker but not in the Docker Compose you'll need to figure out the Docker networking such that you can access the broker. For example you can use `--network` to get it to join the network of your Docker Compose. – Robin Moffatt Mar 03 '22 at 12:46
  • For more details see https://rmoff.net/2018/08/02/kafka-listeners-explained/ and https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/ – Robin Moffatt Mar 03 '22 at 12:47

0 Answers0