How to connect application to kafka pod in k8s?

Question

I have k8s cluster with core-pod(this is my spring-boot application) and kafka pod. There are services for connections to this services, but, when start application pod - there is connectivety issue. I tries to connect this containers in just simple docker and it was success

So, my application yml and pod yml

server:
  port: 8080
logging:
  level:
    nuclear.bot.core: TRACE
notification:
  level: INFO
spring:
  kafka:
    template:
      default-topic: PARSER.EVENT
    consumer:
      bootstrap-servers: kafka-service:9092
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: nuclear.bot.core.config.MessageAdapter
      group-id: core
    properties:
      key-class: java.lang.String
      value-class: nuclear.bot.core.dto.AgentMessage

so application try to connect to kafka-service on port 9092

and my app pod like this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: core
  labels:
    app: core
spec:
  replicas: 1
  selector:
    matchLabels:
      app: core-pod
  template:
    metadata:
      name: core-pod
      labels:
        app: core-pod
    spec:
      containers:
        - name: nuclear-bot-core
          image: vladi15151/nuclear-bot-core:0.9.7
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP

So, kafka which I trying to connect like this (pod and service together)

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  selector:
    app: kafka-broker
  ports:
    - name: kafka-port
      port: 9092
      targetPort: 9092
      nodePort: 30126
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kafka-broker
  name: kafka-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      hostname: kafka-broker
      containers:
        - image: bitnami/kafka
          imagePullPolicy: IfNotPresent
          name: kafka-broker
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper-service:2181"
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://localhost:9092
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://localhost:9092
            # Creates a topic with one partition and one replica.
            - name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
              value: "true"
            - name: KAFKA_CREATE_TOPICS
              value: "PARSEREVENT:1:1"
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"

Kafka successfully started on kluster, but application throw error message when try to start:

2023-06-09 21:53:26.193  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:27.249  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:27.249  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:27.249  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:28.205  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:28.205  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:28.205  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:29.309  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:29.309  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:29.309  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:30.113  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:30.113  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:30.113  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:31.143  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:31.143  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:31.143  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected

What can I try to use more?

Note: `KAFKA_CREATE_TOPICS` does nothing with the bitnami container. Check the docs — OneCricketeer, Jun 10 '23 at 00:45
And `KAFKA_ZOOKEEPER_CONNECT` isn't doing anything since you have no Zookeeper service and bitnami container is using Kraft mode by default — OneCricketeer, Jun 10 '23 at 07:09

score 1 · Answer 1 · answered Jun 10 '23 at 00:44

You've made the broker only allow for connection from itself, not external connections. You need to modify the two listener env variables. Connect to Kafka running in Docker

Then, it would be better if you used fully qualified address kafka-service.svc.cluster.local:9092.

If you use Strimzi, you'd not have this issue.

How to connect application to kafka pod in k8s?

1 Answers1