0

I'm using the Wurstmeister Docker Image to try to run a very basic single broker Kafka server just for testing my application.

It's a simple docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - '2181:2181'
  kafka:
    image: wurstmeister/kafka
    ports:
      - '9092:9092'
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I can enter the container, and create topics using the standard binaries, e.g.

> docker exec -it ${container} kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092
> docker exec -it ${container} kafka-topics.sh --list --bootstrap-server localhost:9092
my-topic

Displaying container information shows the port 9092 exposed:

> docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS         PORTS                                                NAMES
14b0210787cf   wurstmeister/zookeeper     "/bin/sh -c '/usr/sb…"   11 seconds ago   Up 9 seconds   22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   catalog-ingestor-service_zookeeper_1
9ecb59c1bfdd   wurstmeister/kafka         "start-kafka.sh"         11 seconds ago   Up 9 seconds   0.0.0.0:9092->9092/tcp                               catalog-ingestor-service_kafka_1

Using kcat (kafkacat), which I installed using homebrew, I try to connect and get an error:

> kcat -b localhost:9092 -L
%3|1645810235.301|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Failed to resolve 'localhost:9092': nodename nor servname provided, or not known (after 5002ms in state CONNECT)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)

I'm pretty sure I should be trying to connect on localhost:9092, but even when I ping it, it doesn't seem to resolve:

> ping localhost:9092
ping: cannot resolve localhost:9092: Unknown host

Also, I assume I don't need to explicitly create a network / bridge across the Kafka Broker and Zookeeper, since zookeeper:2181 already resolves that bridge?

Greg Peckory
  • 7,700
  • 21
  • 67
  • 114
  • Ping doesn't check ports, only hostnames, and localhost should always resolve.... Your kcat command is correct, though – OneCricketeer Feb 26 '22 at 00:32
  • You may want to see https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker – OneCricketeer Feb 26 '22 at 00:33
  • 1
    Does the error `Failed to resolve 'localhost:9092': nodename nor servname provided` not suggest its a docker port configuration issue as opposed to a Kafka configuration issue? Are there any good troubleshooting tools on Mac (e.g netcat) for this? – Greg Peckory Feb 26 '22 at 14:03
  • The error is saying that the port is included as a part of the hostname. Colons are not allowed in hostnames. Yes, Mac has `nc`, but it won't help diagnose Kafka any more than kcat – OneCricketeer Feb 26 '22 at 15:13
  • Ok, I wasn't talking about the `ping` command, but the `kcat` command (which you said is correct) doesn't even seem to be connecting to the server (`nodenamer or servname not found`) which is what made me think its something to do with the Docker network config. Giving the article on Kafka a read now – Greg Peckory Feb 26 '22 at 16:01
  • Really appreciate the help, btw. – Greg Peckory Feb 26 '22 at 16:01
  • Is there a possibility you've modified `/etc/hosts` file to remove `localhost`? Have you even tried using `kcat -L -b 127.0.0.1:9092` yet? – OneCricketeer Feb 26 '22 at 16:08

2 Answers2

0

Well the ping with a port will not help do check whether you can reach a port or not.

Anyway, I was having a similar issue with Confluent Kafka and I had to explicitly expose the port. See the following yaml:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    restart: always
    expose:
      - '2181'
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: 0.0.0.0:2888:3888
      ZOOKEEPER_MAX_CLIENT_CNXNS: 0
    volumes:
      - /Users/pacone/broker/zookeeper-logs/data:/var/lib/zookeeper/data
      - /Users/pacone/broker/zookeeper-logs/log:/var/lib/zookeeper/log

  kafka-one:
    image: confluentinc/cp-kafka:6.2.0
    container_name: kafka-one
    restart: always
    expose: 
      - '9092'
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
      - '29092:29092'

environment:
  KAFKA_BROKER_ID: 10
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
  KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-one:29092,EXTERNAL://localhost:9092
  KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  KAFKA_DEFAULT_REPLICATION_FACTOR: 1
  KAFKA_NUM_PARTITIONS: 1
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
  KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
volumes:
  - /Users/yourUsername/broker/kafka-logs/data:/var/lib/kafka/data

With previous config I am able to connect from Java: enter image description here

Cheers

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Fran Jimenez
  • 21
  • 2
  • 1
    The port is already exposed if you look at the docker ps output. (expose is not the same as ports) Since you're using a different image, this doesn't really address the question – OneCricketeer Feb 26 '22 at 00:49
  • What's the purpose of the 29092 port? I do think its a Docker config issue as opposed to Kafka – Greg Peckory Feb 26 '22 at 14:07
  • @Greg read the post I linked in the comments, about what "advertised listeners" mean. Keep in mind that your question includes advertised host/port, both of which are deprecated kafka properties – OneCricketeer Feb 26 '22 at 15:16
0

can you check if the 9092 port is getting used by any other process? if yes then try to change it. Also, try to make the docker-compose version 3

Sandhya Ratan
  • 26
  • 3
  • Hi, ideally, you should ask questions in comments not in answers since you might not actually be answering the question. If you cannot comment yet, then please try and increase reputation first. – Luke Briner Mar 03 '22 at 15:57