1

we are planning to introduce AWS spot instances in production ( non-prod is running with spot already ). In order to achieve HA we are running HPA with minimum replicas 2 for all critical deployments. Because of the spot instances behaviour we want to run on-demand instances and one pod should be running on on-demand instances for the same

Question:

Is there anyway i can split pods to get launch one pod of the deployment in on-demand and all the other pods (another one since minimum is 2 and if HPA increase the pods ) of the same deployment in spot instances.

We already using nodeaAffinity and podAntiAffinity since we have multiple node groups for different reasons. Below is the snippet.

        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: category
                operator: In
                values:
                - <some value>
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: <lable key>
                operator: In
                values: 
                - <lable value>
            topologyKey: "kubernetes.io/hostname"
Anil Kumar
  • 381
  • 1
  • 2
  • 9

3 Answers3

1

Short answer is No. No such way to define per replica. As you are already using podAntiAffinity, just by adding the same pod labels, you can ensure no two replicas stays in the same host (if that's not what you are already doing). And then use spotInterruption Handler to drain and reschedule without abrupt downtimes during spot interruptions.

Narain
  • 872
  • 9
  • 11
  • Thanks @narain for info, actually we already using the spotinterruption. Since there is no straight approach, thinking of some hack here. Can we have two deployments with two different names having the same set of labels for both the deployments ( app=myapp) and the service have the selector of same labels (app =myapp), will be trying this not sure whether it works or not. – Anil Kumar Nov 23 '21 at 06:47
  • Hi Anil, Yes that will work. But tying your deployment package based on infrastructure requirement will introduce coupling. But if that works for you, then fine. – Narain Nov 23 '21 at 12:10
  • thanks @narain for the comment. Little confuse about coupling, Can you please let me know little more about it, if any doc/link will be more appreciated. – Anil Kumar Nov 23 '21 at 13:34
  • Hi Anil, Actually no specific document. I am just saying naming deployment different and using same labels to direct traffic. And all these to align between spot and on-demand instance is kind of coupling your cloud setup vs k8s config alignment, which is a kind of coupling IMHO. But like I said if you are Ok and your team is well aware of it, then fine. – Narain Nov 24 '21 at 06:13
  • Sure, got it Narain. Thanks for the heads-up. Will check the two deployments with same label in non-prod then we will update here. Thanks agian. – Anil Kumar Nov 24 '21 at 11:47
1

Following up on your last message:

Will check the two deployments with same label in non-prod then we will update here.

Just wondering how this went. Was there any issues/gotchas from this setup that you could share. Are you currently using this setup, or have you moved on to another setup.

college_coder
  • 11
  • 1
  • Hi @college_coder , for now it is halted , and moved out of this. But will come back with this and see how that goes. Thanks. – Anil Kumar Mar 10 '22 at 17:51
0

I am using affinity and topologySpreadConstraints. The following manifest balances pods between on-demand and spot instances, also on-demand has higher priority. You can play with the maxSkew property, to have more pods on demand than on spot, for example, if you set the value to 2 maxSkew: 2.

One more thing, I don't allow having more than one pod on the same node, this is with the podAntiAffinity parameter.

You will need to set a label for your nodes, in my case I'm using the node-lifecycle node tag which can be on-demand or spot.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-lifecycle
                operator: In
                values:
                - on-demand
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels:
                app: nginx
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: node-lifecycle
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: nginx
mrbit01
  • 181
  • 2
  • 2
  • 9