0

It seems to be good practice to map environments such as dev, qa and production to kubernetes namespaces. To achieve "true" separation, it seems to be a good idea to label nodes exclusively dedicated to one of those namespaces and ensure resources in those environments get scheduled on those nodes only. That's at least our current thinking. There may be manifests one might want to use in those namespaces that should/must not be tampered with. Kubernetes does not seem to support associating namespaces with nodes out of the box. PodNodeSelector admission controller seems close but is not quite what we are looking for. The only option to achieve what we want seems to be a custom mutating admission webhook as documented here.

I guess other people have been here before and there is a solution to address our primary concern which is that we don't want load on dev or qa environments impacting production performance.

Is there any off the shelf solution linking namespaces to nodes?

If not, are there any alternative approaches ensuring that environments do not interfere in terms of load/performance?

Andreas Steffan
  • 6,039
  • 2
  • 23
  • 25
  • Does this answer your question? [How to assign a namespace to certain nodes?](https://stackoverflow.com/questions/52487333/how-to-assign-a-namespace-to-certain-nodes) – nyxgear Nov 29 '22 at 17:45

3 Answers3

2

I guess other people have been here before and there is a solution to address our primary concern which is that we don't want load on dev or qa environments impacting production performance.

Been there, got burned by it.

Multiple environments in one cluster might be a good idea under certain circumstances but mixing dev/qa/stage with production in a single cluster spells trouble. Load itself might not be the main issue, especially if you mitigate effects with proper resource allocation, but any tweak, modification and dev-process induced outage on kube-system pods affects production directly. You can't test updates on kubernetes system components beforehand, any cni issue on dev can slow down or render inoperable production and so on... We went down that path and don't recommend it.

With that being said, separation as such is rather easy. On one of our clusters we do keep dev/qa/stage environments for some projects in single cluster, and separate some of the resources with labels. Strictly speaking not really env-separated but we do have dedicated nodes for elk covering all three environments, separate gitlab runners nodes, database nodes and so on, but principle is same. We label nodes and use nodeAffinity with nodeSelectorTerms to target group of nodes with same label for certain task/service (or environment in your case) separation. As a side notenodeSelector is depricated according to the official documentation.

Const
  • 6,237
  • 3
  • 22
  • 27
0

In my opinion having multiple environments in one cluster is a bad idea, for many reasons.

If you are sure you want to do it, and don't want to kill the performance of production pods, you can easily attach resources to deployments/pods.

Another approach is to attach labels to nodes, and force particular pods to deploy on them using PodNodeSelector

Tomasz Białecki
  • 1,041
  • 6
  • 10
0

In general it is not recommended to use namespaces to separate software environments (dev, test, staging, prod..).

The best practice is to use a dedicated cluster for each environment.

To save on costs, you can take the comprises of using:

  • 1 cluster: dev, test, staging
  • 1 cluster: prod

and with this setup, creation of resources namespace in the cluster shared for dev, testing, and staging get a little more annoying to be managed.

I found it very useful the motivations for using namespaces from the docs.

Still, if you need to ensure a set of nodes is dedicated only to the resources in a namespace, you should use a combination of:

  • podSelector to force scheduling of resources only on nodes of the set
  • and a taint to deny scheduling of any other resource not in the namespace on the nodes of the set

How to create a namespace dedicated to use only a set of nodes: https://stackoverflow.com/a/74617601/5482942

nyxgear
  • 99
  • 1
  • 5