How to setup Kafka Security and listeners in dynamic environment?

Question

These resources provide a great view into how to setup kafka security (encryption + ssl) and good idea of kafka listeners.

setting up ssl auth
listeners and advertised-listeners: a, b, c, d(from confluent guy - Robin M)

But this seems to work well in static environments, what about dynamic environments.
When one uses K8S, and containers (docker with k8s over EC2 etc) or some other orchestration framworks, the hostnames and the number of replicas (even without autoscaling scenario) are not known in advance. So how can we create the certificates and the keystores and the truststores on the fly. And how to configure the listeners values on the fly in server.properties file.

You can make each kafka broker as a service – bigbounty Jul 09 '20 at 06:57 — bigbounty, Jul 09 '20 at 06:57
could you please elaborate a bit. – samshers Jul 09 '20 at 07:09 — samshers, Jul 09 '20 at 07:09

score 1 · Answer 1 · answered Jul 09 '20 at 07:36

This creates a kafka broker but as a load balancer

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/load-balancer-type: Internal
  labels:
    app: kafka
  name: kafka-2-external
  namespace: default
spec:
  ports:
  - name: broker
    port: 9092
    protocol: TCP
    targetPort: kafka
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-2
  sessionAffinity: None
  type: LoadBalancer

How to setup Kafka Security and listeners in dynamic environment?

1 Answers1

Linked