11

I have a HorizontalPodAutoscalar to scale my pods based on CPU. The minReplicas here is set to 5:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-web
  minReplicas: 5 
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

I've then added Cron jobs to scale up/down my horizontal pod autoscaler based on time of day:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: production
  name: cron-runner
rules:
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["patch", "get"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cron-runner
  namespace: production
subjects:
- kind: ServiceAccount
  name: sa-cron-runner
  namespace: production
roleRef:
  kind: Role
  name: cron-runner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-cron-runner
  namespace: production
---

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: django-scale-up-job
  namespace: production
spec:
  schedule: "56 11 * * 1-6"
  successfulJobsHistoryLimit: 0 # Remove after successful completion
  failedJobsHistoryLimit: 1 # Retain failed so that we see it
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: django-scale-up-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa myapp-web --patch '{"spec":{"minReplicas":8}}'
          restartPolicy: OnFailure
----
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: django-scale-down-job
  namespace: production
spec:
  schedule: "30 20 * * 1-6"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 0 # Remove after successful completion
  failedJobsHistoryLimit: 1 # Retain failed so that we see it
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: django-scale-down-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa myapp-web --patch '{"spec":{"minReplicas":5}}'
          restartPolicy: OnFailure

This works really well, except that now when I deploy it overwrites this minReplicas value with the minReplicas in the HorizontalPodAutoscaler spec (in my case, this is set to 5)

I'm deploying my HPA using kubectl apply -f ~/autoscale.yaml

Is there a way of handling this situation? Do I need to create some kind of shared logic so that my deployment scripts can work out what the minReplicas value should be? Or is there a simpler way of handling this?

MDalt
  • 1,681
  • 2
  • 24
  • 46
  • Hello, I'm having a difficulty to understand your question. Could you tell the whole scenario for the time based `HPA` scaling of yours? What exactly you mean by overwriting the `minReplicas` (as it should looking at your example)? You should create `HPA` that would be encompassing both situation (high load/low load) in a single `HPA` basing on the available metrics. – Dawid Kruk Feb 16 '21 at 12:58
  • Ah sorry - I've added the HPA config which should hopefully make it clearer – MDalt Feb 16 '21 at 13:09
  • If you would like to avoid overwriting the value of `replicas` in your `Deployment` (each new `create`/`apply`) you could try to update your `Deployment` with [JSON merge patch](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Is this what you are looking for? – Dawid Kruk Feb 18 '21 at 17:03
  • Ah...you mean each deployment you use a merge patch instead of apply, then this will only update values that have explicitly changed? Will give that a go and revert back if so, sounds smart. – MDalt Feb 18 '21 at 18:41
  • I've had a go with this and clarified the problem. Looks like it's definitely the HPA `minReplicas` value that's overwriting the one set by the CronJob (as opposed to the `replicas` in the Deployment). I tried using JSON merge to deploy the HPA (`kubectl patch -f autoscale.yaml --type=merge -p "$(cat autoscale.yaml)"`) and it didn't work. Any other ideas? – MDalt Feb 19 '21 at 15:00
  • Hello, I think we've had a bit of misunderstanding. I thought you are not changing the `HPA` resource but only the `tag`/`version` of the container during the upgrade process (in the `Deployment`). That's why I suggested going with the `JSON` merge patch for it, where you just add the changes to it and not whole resource. Please take a look on the answer I've posted. – Dawid Kruk Feb 23 '21 at 12:18

1 Answers1

4

I think you could also consider the following two options:

Use helm to manage the life-cycle of your application with lookup function:

The main idea behind this solution is to query the state of specific cluster resource (here HPA) before trying to create/recreate it with helm install/upgrade commands.

I mean to check the current minReplicas value each time before you upgrade your application stack.

Manage the HPA resource separately to application manifest files

Here you can handover this task to a dedicated HPA operator, which can coexist with your CronJobs that adjust minReplicas according specific schedule:

Dawid Kruk
  • 8,982
  • 2
  • 22
  • 45
  • 1
    Great answer. I haven't had chance to try the approaches yet, but will accept for now. Very helpful – MDalt Mar 01 '21 at 19:23