1

I am unable to connect to kafka running in container.

I have .env file

KAFKA_BROKER_ID=1
KAFKA_ENABLE_KRAFT=true
KAFKA_CFG_PROCESS_ROLES=broker,controller
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9094
ALLOW_PLAINTEXT_LISTENER=yes
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,EXTERNAL://:9093,CONTROLLER://:9094
KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://localhost:9093
KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
KAFKA_CFG_BROKER_ID=1
KAFKA_CFG_NODE_ID=1
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
KAFKA_CFG_MESSAGE_MAX_BYTES=5242880
KAFKA_CFG_MAX_REQUEST_SIZE=5242880
BITNAMI_DEBUG=true
KAFKA_CFG_DELETE_TOPIC_ENABLE=true

and running docker in workflow docker run --name kafka-broker --env-file ./container-env.env -d bitnami/kafka:latest I can see kafka server is stated properly.

I am trying to create kafka topic but not able to connect to kafka.


    client = AdminClient(conf={"bootstrap.servers": "localhost:9093"})
    topic_name = "test-topic"
        new_topic = NewTopic(
            topic=topic_name,
            num_partitions=1,
            replication_factor=1,
        )
    client.create_topics(new_topics=[new_topic])
    topic_exists = False
    while not topic_exists:
            cluster_metadata = client.list_topics()
            topics = cluster_metadata.topics
            topic_exists = (new_topic.topic in topics.keys())

Am I missing any environment variable for using wrong ports here? Help would be apreciated

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Panda
  • 513
  • 2
  • 11
  • It looks you are missing the ports with your docker run command to map the port with local host. See more about the bitnami/kafka from [here](https://hub.docker.com/r/bitnami/kafka/) – PRATHEESH PC May 23 '23 at 12:52
  • do you mean docker run command should be like this? docker run -itd -p 9093:9093 --name kafka-broker --env-file ./container-env.env -d bitnami/kafka:latest – Panda May 23 '23 at 13:11
  • Hi Pratheesh, Thank you for helping. It works locally but in workflow(CICD) it does not. Do i need to do anything else to run it in a same way as local? – Panda May 23 '23 at 14:15
  • Use `KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,EXTERNAL://0.0.0.0:9093, CONTROLLER://:9094` to accept "external" connections. Also, you are missing `-p 9093:9093` – OneCricketeer May 23 '23 at 14:19
  • This works fine on locally but At the time of CICD it fails. I am writing integration tests where i am creating kafka docker container as youhave seen in the above code. trying to create kafka topic and read write on the same topic. It fails to create topic and the test case – Panda May 23 '23 at 19:53
  • %3|1684871364.441|FAIL|rdkafka#producer-1| [thrd:localhost:9093/bootstrap]: localhost:9093/bootstrap: Connect to ipv4#127.0.0.1:9093 failed: Connection refused (after 0ms in state CONNECT) – Panda May 23 '23 at 19:57
  • I have this env vars now `KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093,CONTROLLER://:9094 KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://localhost:9093` and -p 9093:9093 in docker run command. It gives an error at the tim of workflow run. @OneCricketeer – Panda May 23 '23 at 20:06
  • For the error `Connect to ipv4#127.0.0.1:9093 failed`... If you're running tests in a container, then 127.0.0.1 is the app container, not Kafka container. You'll need to use `kafka:9092` between two containers. Otherwise, the error explicitly says that Kafka isn't available on port 9093 for localhost. Note that the Kafka container will not start immediately, so you should wait for it... Also, suggest you look at Testcontainers project rather than manually start containers in CI pipeline – OneCricketeer May 24 '23 at 02:27
  • Should I expose port kafka:9092 in this case? – Panda May 24 '23 at 03:03
  • I had used kafka:9092 before but testcase didn’t run it keeps on waiting. But i will modify code to wait for few sec and try run test again – Panda May 24 '23 at 03:06
  • When i use kafka:9092 it gives an error org.apache.kafka.common.KafkaException: Failed to construct kafka producer. – Panda May 24 '23 at 11:43
  • I have asked another question here https://stackoverflow.com/questions/76323281/unable-to-write-on-kafka-topic-created-on-kafka-container – Panda May 24 '23 at 12:21
  • The only reason to expose/forward ports is to access from the host (and use localhost to connect to it). Between two containers, you use the container name, and don't need to expose anything. You need to clarify where tests are running – OneCricketeer May 24 '23 at 13:08
  • Yes I used localhost:9093 as i exposed 9093 port. I created topic successfully using docker exec cmd. But when I run test to write on topic it says no topic present. I want to run tests outside container – Panda May 24 '23 at 20:24

0 Answers0