I'm wondering about the best practices for architecting my Kubernetes clusters. For 1 environment (e.g. production), what organisation should I have in my clusters?
Examples: 1 cluster per technology stack, 1 cluster per exposure area (internet, private...), 1 cluster with everything ... ?
Thanks for your help
I'm not a Kubernetes expert, so I'll give you some generic guidance to help until someone who knows more weighs in.
Solution vs Technology Stack
"Solution" is where you have a number of systems that exist to addresses a specific business problem or domain. This could be functional e.g. finance vs CRM vs HR.
Technology stacks in the literal sense is not likely to be relevant. True, it's not uncommon for different solutions & systems to be comprised of different technology (is that what you were meaning?) - but that's usually a by-product, not the primary driver.
Let's say you have two major solutions (e.g. the finance and CRM). It's likely that you will have situations that impacts one but shouldn't impact the other.
Planned functional changes: e.g. rolling out a major release. Object Orientated programmers and architects have had this nailed for years through designing systems that are cohesive but loosely-coupled (see: Difference Between Cohesion and Coupling), and through stuff like the Stable Dependencies Principle. Having both solutions dependent on the same cluster makes them coupled in that respect, which.
Planned infrastructure changes: e.g. patching, maintenance, resource reallocation, etc.
Unplanned changes: e.g. un-planned outage, security breaches.
Conclusion
Look at what will be running on the cluster(s), and what solutions they are part of, and consider separation along those lines.
The final answer might be a combination of both, some sort of balance between security concerns and solution (i.e. change) boundaries.
The best way would be is to have 1 kubernetes cluster and have the worker nodes in private subnets. You can choose to have the control plane in a public subnet with restricted access like your VPN cidr etc.
If you have multiple teams or application stacks, I'd suggest having different namespaces for each stack as this creates the logical separation of resources.
Also, check the resource limits and quotas that you can apply on kubernetes to prevent over consumption of the resources.
And, as you mentioned multiple application stacks, I am assuming you would have multiple services being exposed for each application or something similar. I would highly recommend using a ingress controller (nginx or anything) to work as single point of entry for each application. You can have more than 1 application listening to 1 load balancer.
Also, have prometheus or ELK monitoring in place as they are great with monitoring k8s components and metrics.
And, I would highly recommend using a tool kubecost and kubebench for enhancing your k8s cluster.
Kubecost is for cost analytics and reporting for k8s components and kubebench would audit your cluster against CIS standards and give you a report on what improvements are required and where.
Please note that the above recommendations are based on best practises and cost efficiency.
