Stop all compute in AKS (Azure Managed Kubernetes)

Question

I have created a managed Kubernetes cluster in Azure, but it's only for learning purposes and so I only want to pay for the compute whilst I'm actually using it.

Is there a easy way to gracefully shut down and start up the VMs, availablity sets and load balancers?

Answer 1

You could use the Azure CLI to stop the the entire cluster:

az aks stop --name myAksCluster --resource-group myResourceGroup

And start it again with

az aks start --name myAksCluster --resource-group myResourceGroup

---

Before this feature, it was possible to stop the virtual machines via Powershell:

az vm deallocate --ids $(az vm list -g MC_my_resourcegroup_westeurope --query "[].id" -o tsv)

Replace MC_my_resourcegroup_westeurope with the name of your resource group that contains the VM(s).

When you want to start the VM(s) again, run:

az vm start --ids $(az vm list -g MC_my_resourcegroup_westeurope --query "[].id" -o tsv)

Answer 2

Only VMs cost money out of all AKS resources (well, VHDs as well, but you cannot really stop those). So you only need to take care of those. Edit: Public Ips also cost money, but you cannot stop those either.

For my AKS cluster I just use portal and issue stop\deallocate command. And start those back when I need them (everything seems to be working fine).

You can use REST API\powershell\cli\various SKDs to achieve the same result in an automated fashion.

Answer 3

Above method (az vm <deallocate|start> --ids $(...)) no longer seems to work.

Solved by first listing the VM scale sets and use these to deallocate/start:

$ResourceGroup = "MyResourceGroup"
$ClusterName = "MyAKSCluster"
$Location = "westeurope"

$vmssResourceGroup="MC_${ResourceGroup}_${ClusterName}_${Location}"

# List all VM scale sets
$vmssNames=(az vmss list --resource-group $vmssResourceGroup --query "[].id" -o tsv | Split-Path -Leaf)

# Deallocate first instance for each VM scale set
$vmssNames | ForEach-Object { az vmss deallocate --resource-group $vmssResourceGroup --name $_  --instance-ids 0}

# Start first instance for each VM scale set
$vmssNames | ForEach-Object { az vmss start --resource-group $vmssResourceGroup --name $_  --instance-ids 0}

Answer 4

There is a new feature just added to AKS:

The AKS Stop/Start cluster feature now in public preview allows AKS customers to completely pause an AKS cluster and pick up where they left off later with a switch of a button, saving time and cost. Previously, a customer had to take multiple steps to stop or start a cluster, adding to operations time and wasting compute resources. The stop/start feature keeps cluster configurations in place and customers can pick up where they left off without reconfiguring the clusters.

https://learn.microsoft.com/en-gb/azure/aks/start-stop-cluster

Answer 5

In your AKS cluster, goto properties and find your Resource group name. search for the Resource group and when you select it, it will list your virtual machines. For each Virtual Machine, select the Operations > Auto-Shutdown option and turn it on. This will turn the VM off saving you money when you aren't developing! To turn them back on again, you will need to follow the advice on previous answers or the answer here

Answer 6

Yes you can do this, I recommend automating this if possible, so you can turn off lower environment clusters overnight and at the weekends, it saves a lot of money.

Here is an example below using Azure Pipelines to turn clusters off and on every weekday at 7AM & 7PM.

pool:
  vmImage: ubuntu-22.04
 
pr: none
 
trigger: none
 
schedules:
- cron: '0 07,19 * * 1,2,3,4,5' # run on weekdays only at 7AM & 7PM.
  displayName: Weekday Start & Stop AKS
  branches:
    include:
    - main
  always: true
 
variables:
  - name: serviceConnection
    value: <SERVICE CONNECTION>
 
  - name: aksResourceGroup
    value: <AKS RESOURCE GROUP>
 
  - name: aksCluster
    value: <AKS CLUSTER NAME>
 
jobs:
- job: StartStopAKS
  displayName: Start/Stop AKS
  steps: 
  - task: AzureCLI@2
    displayName: Get AKS Power Status
    inputs:
      azureSubscription: ${{ variables.serviceConnection }}
      scriptType: pscore
      scriptLocation: inlineScript
      inlineScript: |
        $aksResourceGroup = "$(aksResourceGroup)"
        $aksCluster       = "$(aksCluster)"
        $status           = $(az aks show --resource-group $aksResourceGroup  --name $aksCluster --query agentPoolProfiles[0].powerState.code -o tsv)
 
        Write-Host "##[warning]AKS Cluster $aksCluster is $status"
        Write-Host "##vso[task.setvariable variable=aksStatus;]$status"
 
  - task: AzureCLI@2
    displayName: Power On/Off AKS
    inputs:
      azureSubscription: ${{ variables.serviceConnection }}
      scriptType: pscore
      scriptLocation: inlineScript
      inlineScript: |
        $aksResourceGroup = "$(aksResourceGroup)"
        $aksCluster       = "$(aksCluster)"
        $status           = "$(aksStatus)"
 
        if ($status -eq "Stopped") {
          Write-Host "##[warning]Cluster Status is $status, Starting Cluster..."
          az aks start --resource-group $aksResourceGroup --name $aksCluster --verbose  
        }
        else {
          echo "##[warning]Cluster Status is $status, Stopping Cluster..."
          az aks stop --resource-group $aksResourceGroup --name $aksCluster --verbose
        }

ref: https://jimferrari.com/2023/04/25/auto-shutdown-azure-kubernetes-service-clusters-aks/