How can I expose a Statefulset with a load balancer?

Question

I currently trying to create a cluster of X pods witch each have a personal persistent volume. To do that I've created a StateFulSet with X replicas and a PersistentVolumeClaimTemplate This part is working.

The problem is that it's seem's to be impossible to expose thoses pods with a LoadBalancer in the same way as a deployment (because of the uniqueness of a pods in a statefulset).

At this moment I've tried to expose it as a simple deployment witch is not working and the only way I've found is to expose each pods one by one (I've not tested it but I saw it on this) but it's not that scalable...

I'm not running kubernetes on any cloud provider platform then please avoid exclusive command line.

If pods are so unique - why you run them all as one StatefulSet? You should run them as independent deployments and use Load Balancer as usual. — Vasili Angapov, Apr 24 '19 at 09:26
The pods does the same things, they keep the ethereum blockchain sync (to be more precise) then I only want they persistent storage be unique. And it's not possible to have a volumeClaimTemplate inside a deployment. — yatsukino, Apr 24 '19 at 09:30
Stateful sets are not that scalable as you think. This is mainly due to their unique identity and persistence provision. You might want to create headless service before them and have another service which exposes these pods as normal service. About exposing all of them via loadbalancer, i dont think that's a good idea anyways. — Prateek Jain, Apr 24 '19 at 10:00
Even if it's not a good idea the way I've tried it doesn't work... But in the case that I don't expose them, how can I request information to thoses pods without a loadbalancer ? — yatsukino, Apr 24 '19 at 10:03
As far as I know, sataefulset is not governed by deploy or rs or rc. To connect you definitely need service. I would recommend you to look at https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ — Prateek Jain, Apr 24 '19 at 10:06
Ok the Konstantin Vustin answer is working but I'm not sure to understand why it's not a good idea to expose them? Is there a security issue or something? — yatsukino, Apr 24 '19 at 10:18
Hey @yatsukino, I was in the neighborhood and decided to make it clear for you and somebody else. First of all k8s Service doesn't care what to balance StatefulSet pods or Deployment pods. They are still pods. The Service works via label selector, it selects pods and provides port mappings for creating Endpoints. ClusterIP - is a way to expose always uniq frontend virtual IP:port for balancing between endpoints. The NodePort is a way to expose the same port on every cluster node + ClusterIP — Konstantin Vustin, Aug 14 '19 at 17:15
Now about Headless Service. ClusterIP to None makes a Service headless, i.e. without frontend IP:port. But there are two more options: headless service with label selector or without it. For headless Services that define selectors, the endpoints controller creates Endpoints records in the API, and modifies the DNS configuration to return records (addresses) that point directly to the Pods backing the Service. For headless Services that do not define selectors, the endpoints controller does not create Endpoints records. That's it! — Konstantin Vustin, Aug 14 '19 at 17:21
So, a case of using the headless service for StatefulSet pods is discovery through the DNS records. It could be useful in some cases of development, when you want just get a list of all DNS records and implement your own custom discovery or in case of deploying some types of NoSQL databases in Kubernetes — Konstantin Vustin, Aug 14 '19 at 17:25
So, it definitely not a security question. This is the question of Kubernetes flexibility for its users — Konstantin Vustin, Aug 14 '19 at 17:32

score 11 · Accepted Answer · answered Apr 24 '19 at 10:07

The problem is that it's seem's to be impossible to expose thoses pods with a LoadBalancer in the same way as a deployment (because of the uniqueness of a pods in a statefulset).

Why not? Here is my StatefulSet with default Nginx

$ k -n test get statefulset
NAME      DESIRED   CURRENT   AGE
web       2         2         5d
$ k -n test get pods
web-0                    1/1       Running   0          5d
web-1                    1/1       Running   0          5d

Here is my Service type LoadBalancer which is NodePort (in fact) in case of Minikube

$ k -n test get svc
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
nginx     LoadBalancer   10.110.22.74   <pending>     80:32710/TCP   5d

Let's run some pod with curl and do some requests to ClusterIP:

$ kubectl -n test run -i --tty tools --image=ellerbrock/alpine-bash-curl-ssl -- bash
bash-4.4$ curl 10.110.22.74 &> /dev/null
bash-4.4$ curl 10.110.22.74 &> /dev/null
bash-4.4$ curl 10.110.22.74 &> /dev/null
bash-4.4$ curl 10.110.22.74 &> /dev/null

Let's check out Nginx logs:

$ k -n test logs web-0
172.17.0.7 - - [18/Apr/2019:23:35:04 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.0"
172.17.0.7 - - [18/Apr/2019:23:35:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.0"
172.17.0.7 - - [18/Apr/2019:23:35:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.0"
$ k -n test logs web-1
172.17.0.7 - - [18/Apr/2019:23:35:15 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.0"

172.17.0.7 - is my pod with curl:

NAME                     READY     STATUS    RESTARTS   AGE       IP           NODE
tools-654cfc5cdc-8zttt   1/1       Running   1          5d        172.17.0.7   minikube

Actually ClusterIP is totally enough in case of load balancing between StatefulSet's pods, because you have a list of Endpoints

$ k -n test get endpoints
NAME      ENDPOINTS                     AGE
nginx     172.17.0.5:80,172.17.0.6:80   5d

YAMLs:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: web
  selector:
    app: nginx

Ok wow I was thinking that it was mandatory to set the `clusterIp` value to `None` in the service. But it seem's to work with your trick ! Thanks ! — yatsukino, Apr 24 '19 at 10:14
@yatsukino ClusterIP to None - is the headless service, but it's completely another question — Konstantin Vustin, Apr 24 '19 at 10:20
@yatsukino you can read this here https://stackoverflow.com/questions/52707840/what-exactly-is-a-headless-service-what-does-it-do-accomplish-and-what-are-som?answertab=votes#tab-top — Konstantin Vustin, Apr 24 '19 at 10:22
@KonstantinVustin not sure how this setup is working. The docs mention that Statefulsets are required to work with headless service, can you please explain. doc - https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations — Riddle, Feb 09 '21 at 02:11
@Riddle you are free to use pod label selector as you want. Is pod created by StatefulSet or Deployment it's still just a pod. Topic starter thought it's impossible to expose StatefulSet pods with a LoadBalancer in the same way as a deployment. I just showed it is not. StatefulSet requires Headless Service for other reasons — Konstantin Vustin, Feb 09 '21 at 11:15
ah I see, so I can have two services one of them being a load balancer using pod selectors to route the traffic? — Riddle, Feb 09 '21 at 17:26
@Riddle chatting in the comments it is not proper stackoverflow way. You can ask any question and you'll get an answer — Konstantin Vustin, Feb 10 '21 at 09:28
@KonstantinVustin, agree. posted a question here - https://stackoverflow.com/questions/66466384/can-ingress-controllers-use-select-based-rules — Riddle, Mar 03 '21 at 23:29

How can I expose a Statefulset with a load balancer?

1 Answers1