kubernetes cronjob and updating a secret

Question

Below is my python script to update a secret so I can deploy to kubernetes using kubectl. So it works fine. But I want to create a kubernetes cron job that will run a docker container to update a secret from within a kubernetes cluster. How do I do that? The aws secret lasts only 12 hours to I have to regenerate from within the cluster so I can pull if pod crash etc...

This there an internal api I have access to within kubernetes?

cmd = """aws ecr get-login --no-include-email --region us-east-1 > aws_token.txt"""
run_bash(cmd)


f = open('aws_token.txt').readlines()
TOKEN = f[0].split(' ')[5]


SECRET_NAME = "%s-ecr-registry" % (self.region)


cmd = """kubectl delete secret --ignore-not-found %s -n %s""" % (SECRET_NAME,namespace)
print (cmd)
run_bash(cmd)

cmd = """kubectl create secret docker-registry %s --docker-server=https://%s.dkr.ecr.%s.amazonaws.com --docker-username=AWS --docker-password="%s" --docker-email="david.montgomery@gmail.com" -n %s """ % (SECRET_NAME,self.aws_account_id,self.region,TOKEN,namespace)
print (cmd)
run_bash(cmd)

cmd = "kubectl describe secrets/%s-ecr-registry -n %s" % (self.region,namespace)
print (cmd)
run_bash(cmd)

cmd = "kubectl get secret %s-ecr-registry -o yaml -n %s" % (self.region,namespace)
print (cmd)

https://stackoverflow.com/questions/49654457/how-to-auto-deploy-docker-containers-from-amazon-ecr-to-kubernetes-using-jenkins/50502171#50502171 — Vit, Jan 11 '19 at 11:55

score 10 · Answer 1 · answered May 17 '19 at 17:05

As it happens I literally just got done doing this.

Below is everything you need to set up a cronjob to roll your AWS docker login token, and then re-login to ECR, every 6 hours. Just replace the {{ variables }} with your own actual values.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: {{ namespace }}
  name: ecr-cred-helper
rules:
- apiGroups: [""]
  resources:
  - secrets
  - serviceaccounts
  - serviceaccounts/token
  verbs:
  - 'delete'
  - 'create'
  - 'patch'
  - 'get'

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ecr-cred-helper
  namespace: {{ namespace }}
subjects:
- kind: ServiceAccount
  name: sa-ecr-cred-helper
  namespace: {{ namespace }}
roleRef:
  kind: Role
  name: ecr-cred-helper
  apiGroup: ""

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-ecr-cred-helper
  namespace: {{ namespace }}

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  annotations:
  name: ecr-cred-helper
  namespace: {{ namespace }}
spec:
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          serviceAccountName: sa-ecr-cred-helper
          containers:
          - command:
            - /bin/sh
            - -c
            - |-
              TOKEN=`aws ecr get-login --region ${REGION} --registry-ids ${ACCOUNT} | cut -d' ' -f6`
              echo "ENV variables setup done."
              kubectl delete secret -n {{ namespace }} --ignore-not-found $SECRET_NAME
              kubectl create secret -n {{ namespace }} docker-registry $SECRET_NAME \
              --docker-server=https://{{ ECR_REPOSITORY_URL }} \
              --docker-username=AWS \
              --docker-password="${TOKEN}" \
              --docker-email="${EMAIL}"
              echo "Secret created by name. $SECRET_NAME"
              kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"'$SECRET_NAME'"}]}' -n {{ namespace }}
              echo "All done."
            env:
            - name: AWS_DEFAULT_REGION
              value: eu-west-1
            - name: AWS_SECRET_ACCESS_KEY
              value: '{{ AWS_SECRET_ACCESS_KEY }}'
            - name: AWS_ACCESS_KEY_ID
              value: '{{ AWS_ACCESS_KEY_ID }}'
            - name: ACCOUNT
              value: '{{ AWS_ACCOUNT_ID }}'
            - name: SECRET_NAME
              value: '{{ imagePullSecret }}'
            - name: REGION
              value: 'eu-west-1'
            - name: EMAIL
              value: '{{ ANY_EMAIL }}'
            image: odaniait/aws-kubectl:latest
            imagePullPolicy: IfNotPresent
            name: ecr-cred-helper
            resources: {}
            securityContext:
              capabilities: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: Default
          hostNetwork: true
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: 0 */6 * * *
  successfulJobsHistoryLimit: 3
  suspend: false

I am on bare-metal, and getting an error: `Warning FailedMount 4s (x3 over 5s) kubelet MountVolume.SetUp failed for volume "kube-api-access-g56dv" : object ""/"kube-root-ca.crt" not registered` — Can H. Tartanoglu, Apr 19 '22 at 10:57
In which version of K8s are you having this issue? Did you try to add the "automountServiceAccountToken: false" in the CronJob.spec.jobTemplate.spec.template.spec? It should be at the same indentation level than " serviceAccountName: sa-ecr-cred-helper" This will prevent to mount automatically the kube-api-access volume. If your workload doesn't need to interact with the API, it should workaround your issue. — A.Villegas, Jun 01 '22 at 07:41

score 0 · Answer 2 · answered Feb 16 '23 at 08:40

I add my solution for copying secrets between namespaces using cronjob because this was the stack overflow answer that was given to me when searching for secret copying using CronJob

In the source namespace, you need to define Role, RoleBinding and 'ServiceAccount`

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-user-user-secret-service-account
  namespace: source-namespace 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: demo-user-role
  namespace: source-namespace 
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    # Secrets you want to have access in your namespace
    resourceNames: ["demo-user" ]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: demo-user-cron-role-binding
  namespace: source-namespace 
subjects:
- kind: ServiceAccount
  name: demo-user-user-secret-service-account
  namespace: source-namespace 
roleRef:
  kind: Role
  name: demo-user-role
  apiGroup: ""

and CronJob definition will look like so:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: demo-user-user-secret-copy-cronjob 
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  successfulJobsHistoryLimit: 3
  startingDeadlineSeconds: 10
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: demo-user-user-secret-copy-cronjob
            image: bitnami/kubectl:1.25.4-debian-11-r6
            imagePullPolicy: IfNotPresent
            command:
              - "/bin/bash"
              - "-c"
              - "kubectl -n source-namespace get secret demo-user -o json | \
                jq 'del(.metadata.creationTimestamp, .metadata.uid, .metadata.resourceVersion, .metadata.ownerReferences, .metadata.namespace)' > /tmp/demo-user-secret.json && \
                kubectl apply --namespace target-namespace -f /tmp/demo-user-secret.json"
          restartPolicy: Never 
          securityContext:
            privileged: false
            allowPrivilegeEscalation: true
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            capabilities:
              drop: [ "all" ]
          serviceAccountName: demo-user-user-secret-service-account

In the target namespace you also need Role and RoleBinding so that CronJob in source namespace can copy over the secrets.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: target-namespace 
  name: demo-user-role
rules:
- apiGroups: [""]
  resources:
  - secrets
  verbs:
  - 'list'
  - 'delete'
  - 'create'
  - 'patch'
  - 'get'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: demo-user-role-binding
  namespace: target-namespace 
subjects:
- kind: ServiceAccount
  name: demo-user-user-secret-service-account
  namespace: source-namespace 
roleRef:
  kind: Role
  name: demo-user-role
  apiGroup: ""

In your target namespace deployment you can read in the secrets as regular files.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 1
    ...
    spec:
      containers:
      - name: my-app
        image: [image-name]
        volumeMounts:
          - name: your-secret
            mountPath: /opt/your-secret
            readOnly: true
      volumes:
        - name: your-secret
          secret:
            secretName: demo-user
            items:
              - key: ca.crt
                path: ca.crt
              - key: user.crt
                path: user.crt
              - key: user.key
                path: user.key
              - key: user.p12
                path: user.p12
              - key: user.password
                path: user.password

kubernetes cronjob and updating a secret

2 Answers2

Linked