0

I'm trying to connect kafka, which is running on docker conatiner (on EC2 server).

I cloned wurstmeister/kafka-docker, and edited docker-compose-single-broker.yml file like below.


version: "2"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "8092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 111.11.11.111 # (my EC2 server ip address)
      KAFKA_CREATE_TOPICS: "test-topic:1:1" # topic_name:partition:replica
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I changed port 9092:9092 to 8092:9092, because my EC2 server doesn't allow port over 9000.
Ip 111.11.11.111 is an example.

With this docker-compose, I successfully started kafka and zookeeper container. Then, I used console-producer and console-consumer on same EC2 server to test kafka connection like below.
$ bin/kafka-console-producer.sh --broker-list 111.11.11.111:8092 --topic test-topic
$ bin/kafka-console-consumer.sh --bootstrap-server 111.11.11.111:8092 --topic test-topic --from-beginning

It worked well on my EC2 server, but I couldn't do the same on my local shell. It seems like it doesn't allow access from outside the server. How can I change my docker-compose file to make it possible to access from other server?

j1h00
  • 11
  • 2
  • 1
    Read this: https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/ – Robin Moffatt May 04 '22 at 10:59
  • We don't know your VPC rules or if any public traffic is allowed to your instances, but if you used `kcat -L` like the linked blog shows, I suspect you'll still see port 9092 being returned as part of the bootstrap request, so you need to set `KAFKA_ADVERTISED_PORT` to 8082 as well, then forward that instead (also, you don't need to (re)build the wurstmeister image) – OneCricketeer May 04 '22 at 14:08

0 Answers0