66

Yesterday, I stopped a Helm upgrade when it was running on a release pipeline in Azure DevOps and the followings deployments failed.

I tried to see the chart that has failed with the aim of delete it but the chart of the microservice ("auth") doesn't appear. I used the command «helm list -n [namespace_of_AKS]» and it doesn't appear.

What can I do to solve this problem?

Error in Azure Release Pipeline

2022-03-24T08:01:39.2649230Z Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
2022-03-24T08:01:39.2701686Z ##[error]Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Helm List Helm List

Super Jade
  • 5,609
  • 7
  • 39
  • 61
fjalcaraz
  • 771
  • 1
  • 4
  • 7
  • 5
    `helm ls -a -n {namespace}` will list all releases within a namespace, regardless of status. You can also use `helm ls -aA` instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before) – Blender Fox Mar 24 '22 at 12:50
  • @BlenderFox thank you, your response solved my problem – fjalcaraz Apr 11 '22 at 07:55
  • @kavyasaraboju-MT thank you for your response. I didn't prove it because I coul fix with the previous response – fjalcaraz Apr 11 '22 at 07:56
  • @fjalcaraz no problem. I will add as an answer for others to find – Blender Fox Apr 11 '22 at 08:08

5 Answers5

99

This error can happen for few reasons, but it most commonly occurs when there is an interruption during the upgrade/install process as you already mentioned.

To fix this one may need to, first rollback to another version, then reinstall or helm upgrade again.

Try below command to list

helm ls --namespace <namespace>

but you may note that when running that command ,it may not show any columns with information

Try to check the history of the previous deployment

helm history <release> --namespace <namespace>

This provides with information mostly like the original installation was never completed successfully and is pending state something like STATUS: pending-upgrade state.

To escape from this state, use the rollback command:

helm rollback <release> <revision> --namespace <namespace>

revision is optional, but you should try to provide it.

You may then try to issue your original command again to upgrade or reinstall.

kavyaS
  • 8,026
  • 1
  • 7
  • 19
39

helm ls -a -n {namespace} will list all releases within a namespace, regardless of status.

You can also use helm ls -aA instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before)

Blender Fox
  • 4,442
  • 2
  • 17
  • 30
15

Try deleting the latest helm secret for the deployment and re-run your helm apply command.

kubectl get secret -A | grep <app-name>
kubectl delete secret <secret> -n <namespace>
user20822134
  • 151
  • 1
  • 3
  • I had this problem when a deployment failed due to a repository name clash. I was not able to use rollback. but deleting the secret did it for me! Thanks! – jonnyeom Mar 07 '23 at 01:59
1

My exact error was

Helm upgrade failed: another operation (install/upgrade/rollback) is in progress" or "##[error]Error: UPGRADE FAILED: release <namespace> failed, and has been rolled back due to atomic being set: timed out waiting for the condition

The error has 2 potential solutions: Either delete the stuck deployment, or fix something in deployment pipeline.


Solution #1: Delete the stuck deployment

If this same pipeline has been working until now, it may just be stuck. Time to delete it in the Azure portal!

  1. Go the Azure Portal > Azure CLI (aka the icon called "Cloud Shell")

Azure portal address bar and header with Cloud Shell icon circled

  1. az account show // This will give which subscription are you in.

  2. az account set --subscription

  3. az aks get-credentials --name --resource-group

  4. kubectl get secrets -n

  5. In this situation, the latest release has a problem, so delete that release by running the following:

  6. kubectl delete secret -n sh.helm.release.v1..v88 (or whatever the last release was as shown after last command)

  7. Run the pipeline again, it works :-)


Solution #2: Fix the pipeline

If you've just updated your pipeline, or haven't run it yet, you may have a bug. That was the case for my particular Azure DevOps pipeline.

  1. The azure-pipeline.yml stage is missing some parameters (or whatever the problem is in your pipeline)
  2. Fix it.
  3. Deployment succeeds!

Successful deployment via Azure DevOps pipeline

Super Jade
  • 5,609
  • 7
  • 39
  • 61
0

On my side, I just do the helm rollback xxx. and it is ok. xxx is the app name.

syiAtProvence
  • 41
  • 2