How to auto deploy Docker containers from Amazon ECR to Kubernetes using Jenkins

Question

This is our environment:

1. I have a Kubernetes cluster running on Amazon.
2. Jenkins CI/CD running on Amazon that connects to a private GitLab and builds our services as Docker images.
3. Amazon ECR that stores our Docker images.

My questions:

1. How can I auto deploy images from ECR to Kubernetes (as pods) once the Jenkins pipeline pushes newly built images to ECR?
2. Can I do that within Jenkins pipeline? I've read lots of material but I couldn't find how should it be done.

There is also a 3rd party tool like Keel but it doesn't support Amazon ECR (Webhook problem with ECR).

Any help would be appreciated.

Answer 1

I have a similar workflow and I hope this helps you get some direction. I am using bitbucket pipeline for CI, but I am sure Jenkins will work fine as well.

This is what I do in my CI flow :

• Build my code and install dependencies
• Create a container with a unique tag ( commit-id ) > my-cntnr:12
• Push to ECR
• Curl Rancher API for my-pod > set(image:my-cntnr:12)
• Kubernates updates the pod and pulls the container with tag 12 from ECR

Here is the script for reference :

      - composer install --no-interaction
      - docker build -t cms .
      - docker tag myrepo:latest 123456789.dkr.ecr.my-region.amazonaws.com/myrepo:$BITBUCKET_BUILD_NUMBER
      - aws ecr get-login --no-include-email --region my-region >> login.sh
      - sh login.sh
      - docker push 123456799.dkr.ecr.my-region.amazonaws.com/myrepo:$BITBUCKET_BUILD_NUMBER
      - sh .docker/workload-update.sh // my curl script calling rancher API

note: Since I am using Rancher, I can use Rancher API to update pods and their configuration.

---

Now for the ECR credentials part for Kubernetes, you have to create a secret ( a Kubernetes only entity), this secret is created using your AWS ECR details. Then you can use this secret in your pod.yml as image-pull-secret. This will tell k8 to use the secret and pull the image from ECR

I have a simple script to quickly do that.

#
# RUN me where kubectl is available,& make sure to replace account,region etc
#
ACCOUNT=123456789
REGION=my-region
SECRET_NAME=${REGION}-ecr-registry
EMAIL=email@email.com ( can be anything)

#
# Fetch token (which will expire in 12 hours)
#

TOKEN=`aws ecr --region=$REGION get-authorization-token --output text --query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`

#
# Create or replace registry secret
#

kubectl delete secret --ignore-not-found $SECRET_NAME
kubectl create secret docker-registry $SECRET_NAME \
 --docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
 --docker-username=AWS \
 --docker-password="${TOKEN}" \
 --docker-email="${EMAIL}"

And this is how you can use it in your pod.yml

apiVersion: v1
kind: Pod                                            
metadata:
  name: my-app   
  labels:
    app: my-app                              
spec:                                                
  containers:
    - image: 123456789.dkr.ecr.my-region.amazonaws.com/my-repo
      name: -cntnr                            
      ports:
        - containerPort: 8080    
  imagePullSecrets:
  - name: my-secret-name ( this will be same as name of secret we created earlier)

---

I've written a detailed article about the process as well. Please find it here.

Answer 2

Yes, it is possible. You can use Amazon ECR for your Kubernetes cluster, but you need to create a secret with credentials. You can develop a pipeline for Jenkins which will automatically deploy updates to your Kubernetes cluster. For this, you need to configure a trigger, for example successful completing of the previous pipeline which builds your image. You may use kubernetes-plugin for developing your pipeline, or you can just call kubectl update with a new image tag. You can update your cluster using default update for deployment or 3rd party tool, for example helm.

*updates

There is a good doc: Using AWS EC2 Container Registry. For Jenkins and Kubernetes I can advise you to use kube2iam, it helps to avoid expiring of the authorization tokens. Create custom roles for Kubernetes nodes and for Jenkins, also don’t forget to add permission to assign the roles for kube2iam.

Answer 3

If you want to see how to automate CI/CD with multiple environments on Kubernetes using GitOps for promotion between environments and Preview Environments on Pull Requests you might wanna check out my recent talk on Jenkins X at DevOxx UK where I do a live demo of this on GKE. Though Jenkins X works on AWS, AKS and GKE and other kubernetes clusters too.

When you merge a change to the master branch, Jenkins X creates a new semantically versioned distribution of your app (pom.xml, jar, docker image, helm chart). The pipeline then automates the generation of Pull Requests to promote your application through all of the Environments via GitOps. You can then decide to rollback versions after the apps release pipeline has completed (as there's an environment pipeline too).

Answer 4

Very helpful - thank you for this. One quick addition: if you're using namespaces, you will need to add -n <namespace> to each of the kubectl commands for this to work. I added a NAMESPACE environment variable to the cron job script from your Medium article to handle this.

Example:

ACCOUNT=123456789876
REGION=your-region
NAMESPACE=your-namespace
SECRET_NAME=${REGION}-ecr-registry
EMAIL=dummy.email@email.com
TOKEN=`aws ecr get-login --region ${REGION} --registry-ids ${ACCOUNT} | cut -d' ' -f6`
echo "ENV variables setup done."
kubectl delete secret --ignore-not-found $SECRET_NAME -n $NAMESPACE
kubectl create secret docker-registry $SECRET_NAME -n $NAMESPACE \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
echo "Secret created by name. $SECRET_NAME"
kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"'$SECRET_NAME'"}]}' -n $NAMESPACE
echo "All done."