3

What I'm trying to solve : Have a Java microservice be aware of total number of Replicas. This replica count is dynamic in nature

Problem: Kubernetes downward API has limited metadata that doesnt include this information. Is there a way to qausi-query a kubectl-like command natively from a container?

Why am I trying to do this: In relation to Kafka, new replica will cause a rebalance. Looking to mitigate rebalancing when new containers come online/offline with additional business logic.

Goal: Create an arbiter java-service that detects replica count for deployment xyz and orchestrates corresponding pods to yield on Kafka connection until criteria is met

also if this is crazy, I wont take offense. In that case I'm asking for a friend

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
stackoverflow
  • 18,348
  • 50
  • 129
  • 196
  • 1
    have you went through [this thread](https://stackoverflow.com/questions/61679271/kafka-streams-on-kubernetes-long-rebalancing-after-redeployment)? especially static memberships could suite your use case. – eis Apr 12 '21 at 18:57
  • You could use [KEDA](https://keda.sh/) to partially solve this – OneCricketeer Dec 20 '22 at 20:12

1 Answers1

3

Kubernetes downward API has limited metadata that doesnt include this information. Is there a way to qausi-query a kubectl-like command natively from a container?

You need to query the Kubernetes API server for info about the number of replicas for a specific Deployment. You can do this with kubernetes-client java.

Why am I trying to do this: In relation to Kafka, new replica will cause a rebalance. Looking to mitigate rebalancing when new containers come online/offline with additional business logic.

Sounds like you want consistent number of Pods all the time, e.g. avoiding Rolling Deployment? In your Deployment, you can set strategy: to type: Recreate - then the current Pod will be removed first, and then the new will be created - so at most 1 is running at the same time (or the same number as replicas).

StatefulSet

When you want at most X number of replicas you should consider using StatefulSet as its behavior differ from Deployment when e.g. a Node becomes unreachable. Deployment has the behavior, at least X number of replicas.

Jonas
  • 121,568
  • 97
  • 310
  • 388
  • The strategy type 'Recreate' appears to only take action when a version is changed. When services fall-over/die the pods of the deployment will get recreated one-by-one causing Kafka to rebalance. I was expecting all the pods to come offline at the same time which would have been an ideal solution from a Kafka perspective – stackoverflow Apr 13 '21 at 13:31
  • If you use `StatefulSet` instead of `Deployment` you get that behavior even when the pod/node fail or die. – Jonas Apr 13 '21 at 15:20
  • There are two valid update strategies, RollingUpdate and OnDelete. from https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ – stackoverflow Apr 13 '21 at 17:03
  • StatefulSet has a different behavior when a node becomes unreachable, e.g. if replicas: 2, then it means **at most 2 replicas** but for Deployment it means **at least 2 replicas**. See e.g. https://medium.com/tailwinds-navigator/kubernetes-tip-how-statefulsets-behave-differently-than-deployments-when-node-fails-d29e36bca7d5 – Jonas Apr 13 '21 at 17:24