0

I have a PostgreSQL Kubernetes Service based on Patroni/Spilo. This Kubernetes service deploys a cluster of three PostgreSQL pods + three Etcd pods. During maintenance, I had a failure and I wasn't able to restore the old configuration that worked fine before the rolling update.

I searched for documentation and it seems StatefulSets doesn't support rollbacks as deployment. I found this thread that references this doc.

To be honest, however, I didn't understand how to proceed.

My cluster has the following pods:

postgres-0
postgres-1
postgres-2
etcd-0
etcd-1
etcd-2

my rolling update simply needed to upgrade the etcd image from 3.3.20 to 3.5.1. The upgrade started to update etcd-2 and the pod crashed for several reasons. So my intention was to stop the update and revert the etcd-2 to 3.3.20.

How I should proceed in a situation like this? How liveness and probing can help me here? At the moment, the solution proposed in that thread is a bit confusing to me.

Salvatore D'angelo
  • 1,019
  • 3
  • 14
  • 39

1 Answers1

1

To undo changes that have been made, first checkout the rollout history kubectl rollout history sts <name> -n <namespace if not default>.

Get more details about a revision kubectl rollout history sts <name> --revision <number> -n <namespace if not default>.

Undo the changes kubectl rollout undo sts <name> --to-revision <number> -n <namespace if not default>

gohm'c
  • 13,492
  • 1
  • 9
  • 16
  • I read this in other docs, but if I give the commands above I simply got ```Error from server (NotFound): statefulsets.apps "ccdb-cfdb" not found```, but the stateful sets exists. – Salvatore D'angelo Dec 14 '21 at 09:47
  • These are my sts ```statefulset.apps/ccdb-cfdb statefulset.apps/ccdb-etcd``` – Salvatore D'angelo Dec 14 '21 at 09:47
  • Is there any setting on the server/YAML to enable the revision history tracking? – Salvatore D'angelo Dec 14 '21 at 09:49
  • Have you specify the correct namespace? Updated answer just in case. – gohm'c Dec 14 '21 at 09:51
  • yes, you're right. The problem was the namespace. A really silly problem. Thank you. – Salvatore D'angelo Dec 14 '21 at 12:39
  • When I list the revision of sts, do you know if it is possible to add a comment next to each revision so I can understand its content? – Salvatore D'angelo Dec 14 '21 at 12:43
  • what revision does this command apply? Suppose I have already 5 revisions and I want to comment all of them, how I can do it by specifying the revision? – Salvatore D'angelo Dec 14 '21 at 14:05
  • For StatefulSet, change cause is automatically maintain for you when you use `--record` with kubectl to manipulate your StatefulSet. Every successful changed will also automatically update `metadata.generation` which is the revision number. There's no official way to update these 2 fields by hand. Best bet is to use `kubectl rollout history sts --revision ` to check what fields have changed. – gohm'c Dec 14 '21 at 16:14