Unable to connect from Java to kafka running inside docker container

Question

I am very new to kafka and docker, and I want to connect from Java to Kafka running inside the docker container, but I get an error, and it seems to be a lack of networking knowledge from my side.

I have defined an env variable: KAFKA_HOSTNAME=kafka

Here is my docker-compose.yml file:

version: '1.1'

networks:
  sb:
    driver: bridge

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    hostname: zookeeper
    networks:
     - sb
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    hostname: ${KAFKA_HOSTNAME:-kafka}
    depends_on:
      - zookeeper
    networks:
     - sb
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: ${KAFKA_HOSTNAME:-kafka}
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${KAFKA_HOSTNAME:-kafka}:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

When I run the docker-compose up, I can see that the zookeeper and the Kafka broker are started successfully:

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                                        NAMES
60460c26ef86        confluentinc/cp-kafka:latest       "/etc/confluent/dock…"   19 minutes ago      Up 19 minutes       0.0.0.0:9092->9092/tcp                       kafka
0d1fd4000a83        confluentinc/cp-zookeeper:latest   "/etc/confluent/dock…"   19 minutes ago      Up 19 minutes       2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zookeeper

Here is my simple java program I try to run:

private static final String bootstrapServers = "192.168.0.102:9092";

//create producer properties
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

//create producer
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);

//producer record
ProducerRecord<String, String> record = new ProducerRecord("first_topic", "hello from java");

//send data
producer.send(record);


producer.flush();
producer.close();

In the docker logs I can see that it is creating the first-topic inside the kafka container, but in the Java program I get the following error:

[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error connecting to node kafka:9092 (id: 1 rack: null) java.net.UnknownHostException: kafka

I have a few questions: 1. What am I missing? 2. How should I correctly set the KAFKA_ADVERTISED_HOST_NAME property in docker-compose file? Is it correct to hardcode the IP? 3. How should I correctly set the KAFKA_ADVERTISED_LISTENERS: property?

Thank you very much!

score 3 · Answer 1 · answered Sep 12 '19 at 18:56

3

if you are running on mac try KAFKA_HOSTNAME=docker.for.mac.localhost

on windows try KAFKA_HOSTNAME=docker.for.win.localhost

answered Sep 12 '19 at 18:56

funtoos

295
1
4
17

the same unknown host exception: kafka – John P Sep 12 '19 at 19:05
@AndrewT Because you are setting the advertised name to `kafka` – OneCricketeer Sep 12 '19 at 19:15
so how would it be correct to map it? – John P Sep 12 '19 at 19:21
1

@AndrewT If you want your machine `localhost` to continue to sent requests to `localhost`, then the container needs to "advertise" as `localhost`. Same logic can be applied to an IP address or host name. But `kafka` (the container service name) isn't known to your external DNS server, so that is why it'll fail unless your producer code is also in a container. – OneCricketeer Sep 12 '19 at 20:23

Unable to connect from Java to kafka running inside docker container

1 Answers1