Run kafka in docker which is not the part of kubernetes cluster but access it from that cluster

Question

And i attempted to create kafka in dockerized environment, but not the part of kubernetes cluster

i have a trouble in accessing kafka bootstrap server inside of the kubernetes pod.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "29092:29092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,INTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

above code is docker-compose file of the kafka that i collected in google.

well, it works well in my Mac Environment by using consume tools, Python and Java

But the problem occurs in kubernetes cluster and dockerized environment(not the same with kafka container).

when i run kafkacat test on kubernetes pod with linux, like

kafkacat -P -b host.docker.internal:29092 -t test

but

%3|1681306449.415|FAIL|rdkafka#producer-1| [thrd:localhost:29092/1001]: localhost:29092/1001: Connect to ipv4#127.0.0.1:29092 failed: Connection refused (after 1ms in state CONNECT)
%3|1681306449.511|FAIL|rdkafka#producer-1| [thrd:localhost:29092/1001]: localhost:29092/1001: Connect to ipv4#127.0.0.1:29092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

occurs. or when i tried to connect with docker, Using port 9092

%3|1681307275.048|FAIL|rdkafka#producer-1| [thrd:172.17.0.1:9092/bootstrap]: 172.17.0.1:9092/bootstrap: Connect to ipv4#172.17.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)

Or 29092

org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1001 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.

I tried many things that i can like setting advertised.listeners to localhost, external ip

OneCricketeer · Accepted Answer · 2023-04-12T18:56:59.903

0

attempted to create kafka in dockerized environment, but not the part of kubernetes cluster

Well, you could do that using Strimzi

Otherwise, Kubernetes isn't the problem as long as Egress rules allow external connections.

You have INTERNAL which is only reachable from containers in the Compose network. You have EXTERNAL_SAME_HOST, which as the name implies, are only reachable from services on that same host machine, not external Kubernetes pods because localhost is returned, meaning the pods try to connect to themselves.

You need to advertise 192.168.1.x, or whatever LAN/WAN IP or hostname that can be reached from Kubernetes hosts to your own development machine.

If you want to use host.docker.internal

edited Apr 12 '23 at 18:56

answered Apr 12 '23 at 18:51

OneCricketeer

179,855
19
132
245

thx a lot. i solved issue by using strimzi. But what i worry is that does kafka cluster in kubernetes eat many sources that my computer has? i searched some issues related to that but i want to know about your personal view for my reference. – 김민석 Apr 13 '23 at 14:02
Not much more resources than outside of kubernetes. It's still a container, after all. The only overhead is what Strimzi adds, such as the operator, metrics exporters, etc – OneCricketeer Apr 13 '23 at 14:13

Run kafka in docker which is not the part of kubernetes cluster but access it from that cluster

1 Answers1