0

I'm trying to create a kubernetes deployment file that is able to connect to another node that is running mongoDB. This is working already.

Now I also want to connect to my Kafka container running locally in docker. I'm not able to get this working.

How can I solve this, what should my bootstrap servers input be?

I'm also new to Kubernetes :)

I tried using my ipv4 address of my local machine as "bootstrap servers" but then I get connection refused.

This is my deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tweet-service-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tweet-service-deployment
  template:
    metadata:
      labels:
        app: tweet-service-deployment
    spec:
      containers:
      - name: tweet-service-deployment
        image: dirkl1/kwebbel-tweet-service:latest
        ports:
          - containerPort: 80
        env:
          - name: ASPNETCORE_URLS
            value: http://+:80
          - name: MongoDB__USER
            valueFrom:
                  secretKeyRef:
                    name: mongo-secret
                    key: mongo-user
          - name: MongoDB__PASSWORD
            valueFrom: 
                  secretKeyRef:
                    name: mongo-secret
                    key: mongo-password
          - name: MongoDB__HOST
            valueFrom: 
                configMapKeyRef:
                  name: mongo-config
                  key: mongo_url
          - name: Kafka__BootstrapServers
            valueFrom:
                configMapKeyRef:
                  name: kafka-config
                  key: bootstrap.servers
---          
apiVersion: v1
kind: Service
metadata:
  name: tweet-service
spec:
  type: NodePort
  selector:
    app: tweet-service-deployment
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30200
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Dirk lemmen
  • 13
  • 2
  • hey @dirk - where (platform) is this running? you mention node(s) and also kafka running locally - can you specify where are those nodes please? is this external like EKS or local like minikube? Cheers! – marianogg9 Mar 28 '23 at 14:39
  • I wouldn't try to run things across multiple container environments in most cases. In particular, the remote Kubernetes can't connect back to your local Docker. Can you set up Kafka in Kubernetes as well, either hand-writing a StatefulSet for it or using a prebuilt Helm chart? – David Maze Mar 28 '23 at 16:16
  • You should share your ConfigMap values as part of a [mcve] – OneCricketeer Mar 28 '23 at 22:02
  • @marianogg9 it is running on a Kubernetes cluster from Docker itself when enabling Kubernetes in Docker desktop. – Dirk lemmen Mar 29 '23 at 07:31
  • @DavidMaze I'm using confluent Kafka in my .NET API. Can I just deploy a Apache Kafka with Strimzi and would it still work? – Dirk lemmen Mar 29 '23 at 07:33

1 Answers1

0

tried using my ipv4 address of my local machine as "bootstrap servers" but then I get connection refused

By default, Kafka won't accept connections outside of localhost. K8s runs on a separate network interface, most likely.

want to connect to my Kafka container running locally

Assuming you are using Docker Desktop, then host.docker.internal might work. See answer here for configuring Kafka. Otherwise, your host's LAN IP should work as well, but you still need to modify server.properties file.

However, I would recommend not doing this. You can run Kafka on k8s easily following https://strimzi.io or other Helm Charts available online.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245