3

In Kubernetes, we have multiple environments, separated by different namespace. I want to ensure that one set of nodes are only used by a specific namespace/environment.

  1. Nodes of specific labels should reject all pods, which doesn't belong to a namespace
  2. Pods from a specific namespace should always choose nodes with the labels configured.

What is the way to achieve it? Heard about mutating webhook admission controllers, Anybody has a sample, to see how it works.

Ysak
  • 2,601
  • 6
  • 29
  • 53
  • Does this answer your question? [How to assign a namespace to certain nodes?](https://stackoverflow.com/questions/52487333/how-to-assign-a-namespace-to-certain-nodes) – nyxgear Nov 29 '22 at 17:35

3 Answers3

5

You can use mutating webhook to mutate an incoming pod request from a specific namespace to add node affinity or node selector in the pod spec.

An example of nodeSelector admission controller here. Full guide on how to use it.

Prepare API Server

Assuming your cluster has been deployed with kubeadm, the kube-apiserver.yaml file is the configuration manifest for the Kubernetes API server. It is located in /etc/kubernetes/manifests. Add the PodNodeSelector admission controller in the --admission-control= flag

Then add a label to a node

kubectl label node kubeprod01 env=production

Then use that label in the annotation of the namespace.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: env=production

An example of adding node affinity via mutating webhook.

func mutatePods(ar v1beta1.AdmissionReview, o *options) *v1beta1.AdmissionResponse {
    var reviewResponse = &v1beta1.AdmissionResponse{
        Allowed: true,
    }

    podResource := metav1.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"}
    if ar.Request.Resource != podResource {
        glog.Errorf("expect resource to be %s", podResource)
        return nil
    }

    raw := ar.Request.Object.Raw
    pod := v1.Pod{}
    // glog.V(2).Infof("Object: %v", string(raw))
    if err := json.Unmarshal(raw, &pod); err != nil {
        glog.Error(err)
        return nil
    }

    addPodAffinityPatch := fmt.Sprintf(`[
         {"op":"add","path":"/spec/affinity","value":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"%s","operator":"NotIn","values":["%s"]}]},"weight":1}]}}}
    ]`, o.PodAffinityKey, o.PodAffinityValue)

    glog.V(2).Infof("patching pod")
    reviewResponse.Patch = []byte(addPodAffinityPatch)
    pt := v1beta1.PatchTypeJSONPatch
    reviewResponse.PatchType = &pt

    return reviewResponse
}
Arghya Sadhu
  • 41,002
  • 9
  • 78
  • 107
  • I've recently open-sourced a mutating webhook that does that out of the box: https://github.com/idgenchev/namespace-node-affinity :) – Ivan Genchev Apr 14 '21 at 06:29
2

It can be done via node-selector annotation in the namespace, see an example here

Anton Matsiuk
  • 662
  • 3
  • 8
  • This approach is described in details here https://stackoverflow.com/a/62938949/151641 but I don't think it is sufficient without taints. – mloskot Mar 01 '23 at 11:22
0

To ensure a set of nodes is dedicated only to the resources in a namespace, you should use a combination of:

  • podSelector to force scheduling of resources only on nodes of the set
  • and a taint to deny scheduling of any other resource not in the namespace on the nodes of the set

How to create a namespace dedicated to use only a set of nodes: https://stackoverflow.com/a/74617601/5482942

Also, I found it useful to read the motivations for using namespaces.

Indeed, in general, it is not recommended to have multiple environments (dev, test, staging, prod) in the same k8s cluster.

The best practice is to use dedicated clusters for each environment.

To save on costs, you can take the comprises of using:

  • 1 cluster: dev, test, staging
  • 1 cluster: prod
nyxgear
  • 99
  • 1
  • 5