How to customize the behavior of the kube-scheduler on AKS?

Question

I'm trying to customize the behavior of the kube-scheduler on an AKS cluster (kubernetes v1.19.3), as described in Scheduler Configuration.

My goal is to use the NodeResourcesMostAllocated plugin in order to schedule the pods using the least number of nodes possible.

Consider the following file - most-allocated-scheduler.yaml

apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
  - schedulerName: most-allocated-scheduler
    plugins:
      score:
        disabled:
        - name: NodeResourcesLeastAllocated
        enabled:
        - name: NodeResourcesMostAllocated
          weight: 2

According to the documentation, I can specify scheduling profiles by running something like:

kube-scheduler --config most-allocated-scheduler.yaml

But where exactly can I find the kube-scheduler in order to run the above command? I'd like to do this ideally on a pipeline. Is it possible to do such thing when using AKS?

Answer 1

kube-scheduler is a part of kubernetes control plane. It's components are scheduled on master node, to which on managed kubernetes solutions such as AKS, GKE or EKS, you have no access.

This means it's not possible to reconfigure your kube-scheduler on a running AKS cluster. Compare with this answer on AKS's GitHub page.

However, it is possible to provide custom configuration for your kube-scheduler when creating a new cluster, using cluster definitions, specifically in schedulerConfig section:

schedulerConfig

schedulerConfig declares runtime configuration for the kube-scheduler daemon running on all master nodes. Like kubeletConfig, controllerManagerConfig, and apiServerConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom apiserver config:

"kubernetesConfig": {
    "schedulerConfig": {
        "--v": "2"
    }
}

See here for a reference of supported kube-scheduler options.

...

Keep in mind however that not all options are supported. Docs says that e.g. --kubeconfig is not supported, but as you can read here, this flag is deprecated anyway. There is nothing about --config flag so you can simply try if it works.

You can also achieve it by using Custom YAML for Kubernetes component manifests:

Custom YAML specifications can be configured for kube-scheduler, kube-controller-manager, cloud-controller-manager and kube-apiserver in addition to the addons described above. You will need to pass in a base64-encoded string of the kubernetes manifest YAML file to KubernetesComponentConfig["data"] . For example, to pass a custom kube-scheduler config, do the following:

"kubernetesConfig": {
    "schedulerConfig": {
            "data" : "<base64-encoded string of your k8s manifest YAML>"
        }
}

NOTE: Custom YAML for addons is an experimental feature. Since Addons.Data allows you to provide your own scripts, you are responsible for any undesirable consequences of their errors or failures. Use at your own risk.

So as you can see, even in managed kubernetes solutions such as AKS, kube-scheduler can be customized to certain extent, but only when you create a new cluster.