0

I am attempting to create a simple producer and consumer with two Python scripts, using Kafka deployed on Microk8s. However, when running the producer.py script, I get the following error on repeat:

...|FAIL|rdkafka#producer-1| [thrd:...:9092/bootstrap]: ...:9092/bootstrap: Connect to ipv4#localhost:9092 failed: Connection refused (after 0ms in state CONNECT, ... identical error(s) suppressed

I am fairly confident that this issue is a result of the listeners not being configured correctly, but I have so far been unable to figure out what I need to do to fix them, due to what I assume is my complete lack of any knowledge in this area. I have reviewed these resources, in addition to several others from this site, but have been unable to find a solution, or at least a solution I can understand enough to act upon.

Steps to Reproduce:

The Python scripts to generate the producer and consumer can be found here.

For Microk8s installation, I followed these instructions. I also installed Helm, since my project requirements dictate that I use Helm charts.

I then installed Kafka using:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka-release bitnami/kafka
wb1210
  • 50
  • 8
  • 1
    Well, show us what advertised listeners are being applied to the brokers – OneCricketeer Jan 24 '23 at 17:29
  • Today I think I've learned a little more about the issue, and have followed the instructions for "Accessing brokers from outside the cluster" at: https://github.com/bitnami/charts/tree/main/bitnami/kafka#accessing-kafka-bokers-from-outside-the-cluster , both the load balancer and the NodePort options. However, running "kubectl describe pod/kafka-0 | grep KAFKA_CFG_ADVERTISED_LISTENERS" returns nothing – wb1210 Jan 26 '23 at 03:24
  • For NodePort option B, I tried at various times 30001 and 31111. – wb1210 Jan 26 '23 at 03:25
  • 1
    Unclear why you're grepping a pod. You'd connect to a Service - https://github.com/bitnami/charts/blob/main/bitnami/kafka/templates/svc-external-access.yaml#L1-L10 – OneCricketeer Jan 26 '23 at 15:50
  • Ah, okay! I just tried doing the same command, but substituting pod with service and listing every kafka service that shows up when I use "kubectl get services". Unfortunately, it didn't show anything. I assumed it would show up for kafka-0-external, but no dice. – wb1210 Jan 26 '23 at 17:00
  • 1
    You need to set `externalAccess.enabled: true` in the `values.yaml`, as the readme says – OneCricketeer Jan 26 '23 at 18:49
  • I should probably note that the command I entered above that uses pods provides , when used with bitnami Kafka without any modifications at all, "INTERNAL://$(MY_POD_NAME).kafka-base-headless.default.svc.cluster.local:9093,CLIENT://$(MY_POD_NAME).kafka-base-headless.default.svc.cluster.local:9092" – wb1210 Jan 26 '23 at 18:54
  • externalAccess.enabled is indeed set to true. – wb1210 Jan 26 '23 at 18:55
  • If it helps, this might make a difference: I'm deploying everything on an Ubuntu VM using VirtualBox. – wb1210 Jan 26 '23 at 20:15
  • 1
    Virtualbox has its own network port forwarding that needs done for accessing the k8s cluster from the host. Kubernetes won't manage that – OneCricketeer Jan 26 '23 at 23:44

1 Answers1

1

The Python code in the linked post uses 'localhost:9092', as the error also shows - Connect to ipv4#localhost:9092 failed

If you are trying to run that code in a k8s pod, then you need to give the external broker DNS addresses, not the local pod address.

If you run the Python code from outside the k8s cluster, you need to expose a ClusterIP / NodePort external service or Ingress (as the linked Strimzi post shows; plus, you can can still use Strimzi Operator with Helm, so you don't really need the Bitnami Charts).

At a high level, the advertisted.listeners tells clients how to connect to a specific broker. If you advertise localhost, the pod will try to connect to itself, even if the bootstrap connection worked (setup by just listeners). If you advertise kafka.svc.cluster.local, then it will try to connect to the kafka service in the default namespace... But you still need to actually set boostrap.servers = kafka.svc.cluster.local:9092, for example.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks for your help! I am in the process of attempting to use Strimzi instead of Bitnami. I started a strimzi-kafka-operator pod using these instructions: https://strimzi.io/blog/2018/11/01/using-helm/ . However, it's hanging in Pending status. To expose kafka using external ports, I attempted to follow the Strimzi post, but I don't understand what file I need to edit. – wb1210 Jan 26 '23 at 03:34
  • 1
    That post is 4+ years old, now. Refer latest docs here https://strimzi.io/docs/operators/latest/deploying.html#deploying-cluster-operator-helm-chart-str, with sections below for setting up clients – OneCricketeer Jan 26 '23 at 15:45
  • The new instructions are much-appreciated, but I don't have a cluster to use for the instructions. – wb1210 Jan 26 '23 at 17:10
  • 1
    Huh? Your question said you are using microk8s, which is "a cluster" – OneCricketeer Jan 26 '23 at 18:48
  • Ah, gotcha'. Well, I got the cluster name as microk8s, but when using that as my cluster, I got: "WARN Couldn't resolve server microk8s-kafka-bootstrap:9092 from bootstrap.servers as DNS resolution failed for microk8s-kafka-bootstrap..." – wb1210 Jan 26 '23 at 19:00
  • 1
    Not really clear where your Kafka clients are running, but if they are not also in Virtualbox, on the same VM network, then there is no DNS name for "microk8s-kafka-bootstrap". This is where you need a real DNS server, or edit the /etc/hosts file – OneCricketeer Jan 26 '23 at 23:49
  • I figured out that it defaults to "my-cluster" – wb1210 Jan 27 '23 at 16:25