Scheduled restarts on kubernetes using terraform

Question

I run a kubernetes cluster on aws managed by terraform. I'd like to automatically restart the pods in the cluster at some regular interval, maybe weekly. Since the entire cluster is managed by terraform, I'd like to run the auto restart command through terraform as well.

At first I assumed that kubernetes would have some kind of ttl for its pods, but that does not seem to be the case.

Elsewhere on SO I've seen the ability to run auto restarts using a cron job managed by kubernetes (eg: How to schedule pods restart). Terraform has a relevant resource -- the kubernetes_cron_job -- but I can't fully understand how to set it up with the permissions necessary to actually run.

Would appreciate some feedback!

Below is what I've tried:

resource "kubernetes_cron_job" "deployment_restart" {
  metadata {
    name = "deployment-restart"
  }
  spec {
    concurrency_policy            = "Forbid"
    schedule                      = "0 8 * * *"
    starting_deadline_seconds     = 10
    successful_jobs_history_limit = 10
    job_template {
      metadata {}
      spec {
        backoff_limit              = 2
                active_deadline_seconds      = 600
        template {
          metadata {}
          spec {
                        service_account_name = var.service_account.name
            container {
              name    = "kubectl"
              image   = "bitnami/kubectl"
              command = ["kubectl rollout restart deploy"]
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_role" "deployment_restart" {
  metadata {
    name      = "deployment-restart"
  }

  rule {
    api_groups = ["apps", "extensions"]
    resources  = ["deployments"]
    verbs      = ["get", "list", "patch", "watch"]
  }
}

resource "kubernetes_role_binding" "deployment_restart" {
  metadata {
    name      = "deployment-restart"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.deployment_restart.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = var.service_account.name 
    api_group = "rbac.authorization.k8s.io"
  }
}

This was based on a combination of Granting RBAC roles in k8s cluster using terraform and How to schedule pods restart.

Currently getting the following error: Error: RoleBinding.rbac.authorization.k8s.io "deployment-restart" is invalid: subjects[0].apiGroup: Unsupported value: "rbac.authorization.k8s.io": supported values: ""

So, you are getting an error message, when running it? Or what are you missing? — Fritz Duchardt, Nov 11 '21 at 08:04
@FritzDuchardt updated with what I tried and what error I'm getting — theahura, Nov 11 '21 at 14:18

score 0 · Answer 1 · answered Nov 12 '21 at 09:27

As per the official documentation rolebinding.subjects.apiGroup for Service Accounts should be empty.

kubectl explain rolebinding.subjects.apiGroup

KIND: RoleBinding VERSION: rbac.authorization.k8s.io/v1

FIELD: apiGroup

DESCRIPTION: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.

Scheduled restarts on kubernetes using terraform

1 Answers1