26

I am running the below yaml script to build docker images and push into kubernetes cluster but at the same time I wanted to enable docker layer caching in the azure DevOps while building the yaml script.Could you please explain how to enable or how to add the task in azure devops to do this.

Yaml:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  tag: 'web'
  DockerImageName: 'boiyaa/google-cloud-sdk-nodejs'


steps:
- task: Docker@2
  inputs:
    command: 'build'
    Dockerfile: '**/Dockerfile'
    tags: 'web'
  
- script: |
    echo ${GCLOUD_SERVICE_KEY_STAGING} > ${HOME}/gcp-key.json
               gcloud auth activate-service-account --key-file ${HOME}/gcp-key.json --project ${GCLOUD_PROJECT_ID_STAGING}
               gcloud container clusters get-credentials ${GCLOUD_PROJECT_CLUSTER_ID_STAGING} \
        --zone ${GCLOUD_PROJECT_CLUSTER_ZONE_STAGING} \
        --project ${GCLOUD_PROJECT_ID_STAGING}
  displayName: 'Setup-staging_credentials'


- bash: bash ./deploy/deploy-all.sh staging
  displayName: 'Deploy_script_staging'
Daniel Mann
  • 57,011
  • 13
  • 100
  • 120
PRAVEEN PDBR
  • 423
  • 1
  • 9
  • 25

7 Answers7

54

Here's how I fixed this. I just pull the latest version of the image from my registry (Azure Container Registry in my case) to the Azure DevOps hosted agent. Then I add --cache-from to the Docker build arguments pointing to this latest tag which it just downloaded to the local machine/cache.

- task: Docker@2
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    command: 'login'

- script: "docker pull $(ACR_ADDRESS)/$(REPOSITORY):latest"
  displayName: Pull latest for layer caching
  continueOnError: true # for first build, no cache

- task: Docker@2
  displayName: build
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    repository: '$(REPOSITORY)'
    command: 'build'
    Dockerfile: './dockerfile '
    buildContext: '$(BUILDCONTEXT)'
    arguments: '--cache-from=$(ACR_ADDRESS)/$(REPOSITORY):latest' 
    tags: |
      $(Build.BuildNumber)
      latest

- task: Docker@2
  displayName: "push"
  inputs:
    command: push
    containerRegistry: "$(ContainerRegistryName)"
    repository: $(REPOSITORY) 
    tags: |
      $(Build.BuildNumber)
      latest
Julian
  • 33,915
  • 22
  • 119
  • 174
Kees Schollaart
  • 726
  • 1
  • 5
  • 4
  • This works perfect. You can also use the docker task. – Michael Feb 09 '21 at 12:27
  • 1
    You are a genius! As a side note, this works also with docker-compose, you just need to specify cache_from under build element in the docker-compose.yaml. – Stefan Iancu Apr 17 '21 at 10:00
  • Perfect!!! I was using cache-from without pulling the latest image firstly. You helped me a lot :D – fabiohbarbosa Aug 21 '21 at 10:14
  • Unfortunately, on my end despite pulling the latest image successfully, it still builds all the stages that should have been cached. I opened a question regarding it here: https://stackoverflow.com/questions/69782647/work-around-for-docker-layer-caching-not-working-in-azure-pipeline – cjones Oct 30 '21 at 22:36
  • 1
    Great answer. To make it work for me though, I had to enable buildkit (as mentioned in the [docker documentation](https://docs.docker.com/engine/reference/commandline/build/)) by adding `DOCKER_BUILDKIT=1` before my `docker build` command. – Tyson Williams Jan 23 '23 at 13:44
  • @TysonWilliams, where exactly do we add the DOCKER_BUILDKIT=1. Do you have any reference – Mohamed Imran Feb 04 '23 at 08:34
  • The reference I included links to [the BuildKit documentation](https://docs.docker.com/build/buildkit/#getting-started). Change `docker build ...` to `DOCKER_BUILDKIT=1 docker build ...`. – Tyson Williams Feb 05 '23 at 13:52
  • will this work with multi stag docker images? as layers are not cached – Hany Habib Aug 14 '23 at 09:22
13

Docker layer caching is not supported in Azure DevOps currently. The reason is stated as below:

In the current design of Microsoft-hosted agents, every job is dispatched to a newly provisioned virtual machine. These virtual machines are cleaned up after the job reaches completion, not persisted and thus not reusable for subsequent jobs. The ephemeral nature of virtual machines prevents the reuse of cached Docker layers.

However:

  1. Docker layer caching is possible using self-hosted agents. You can try creating your on-premise agents to run your build pipeline.

    You may need to disable the Job's option 'Allow scripts to access the OAuth token'. For $(System.AccessToken) is passed to docker build using a --build-arg ACCESS_TOKEN=$(System.AccessToken), and its value varies for every run, which will invalidate the cache.

  2. You can also you use Cache task and docker save/load commands to upload the saved Docker layer to Azure DevOps server and restore it on the future run. Check this thread for more information.

  3. Another workaround as described in this blog is to use --cache-from and --target in your Dockerfile.

If the above workarounds are not satisfying, you can submit a feature request to Microsoft Develop Team. Click Suggest a Feature and choose Azure DevOps.

Palec
  • 12,743
  • 8
  • 69
  • 138
Levi Lu-MSFT
  • 27,483
  • 2
  • 31
  • 43
10

Edit: as pointed out in the comments, this feature is actually available without BuildKit. There's an example here on how to use a Docker image as the cache source during a build.

By adding the variable DOCKER_BUILDKIT: 1 (see this link) to the pipeline job and installing buildx, I managed to achieve layer caching by storing the cache as a separate image. See this link for some basics

Here's an example step in Azure DevOps

- script: |
    image="myreg.azurecr.io/myimage"
    tag=$(Build.SourceBranchName)-$(Build.SourceVersion)
    cache_tag=cache-$(Build.SourceBranchName)

    docker buildx create --use
    docker buildx build \
      -t "${image}:${tag}"
      --cache-from=type=registry,ref=${image}:${cache_tag}\
      --cache-to=type=registry,ref=${image}:${cache_tag},mode=max \
      --push \
      --progress=plain \
      .
  displayName: Build & push image using remote BuildKit layer cache

This of course will require each run to download the image cache, but for images that have long-running installation steps in the Docker build process this is definitely faster (in our case from about 8 minutes to 2).

Sebastian N
  • 759
  • 7
  • 13
  • 4
    not sure why a) buildkit is required for that nor buildx. You can simply pull your target image from the reg. before you build and use cache-from. so if we build `fooimg:tag`, we do a `docker pull fooimg:tag` before building, then a `docker build --cache-from fooimg:tag . -t fooimg:tag` – Eugen Mayer Feb 22 '20 at 15:50
  • Thanks for sharing. Didn't know they added this feature to the regular Docker Engine. – Sebastian N Feb 24 '20 at 15:56
  • I'm honestly surprised I haven't heard about this before - the original thread on Github has no mention of using --cache-from: https://github.com/microsoft/azure-pipelines-tasks/issues/6439 The thread on ACR layer caching (https://github.com/Azure/acr/issues/204) mentioned buildx, so after reading the docs for buildx I figured the caching mechanism is what we needed. – Sebastian N Feb 24 '20 at 15:58
  • 2
    be sure to notice that using BUILDKIT:1 will require you to add `--build-arg BUILDKIT_INLINE_CACHE=1` to your build args to support caching layers to the remote registry. – Eugen Mayer Feb 24 '20 at 18:02
  • 2
    Just `--cache-from` on its own won't work on multi-staged builds. Indeed, you need to inline cache in final image for multi-staged builds to benefit from cache on MS hosted agents. – igor Jun 09 '21 at 20:24
3

It looks like Microsoft introduced Pipeline Caching for Azure Devops a while ago and it's possible to cache docker layers. See this link.

Tunay
  • 31
  • 2
  • 1
    I think this only caches the final docker image, not the intermediate layers, so it doesn't actually speed up a Docker build (Docker will still try to rebuild the intermediate layers). – jidicula Sep 15 '21 at 17:42
3

You can also set up "local" Docker layer caching in your Pipeline VM if you don't want to push up your cache to a container registry. You'll need the following steps:

- task: Docker@2
  displayName: Login to ACR
  inputs:
    command: login
    containerRegistry: $(SERVICE_NAME)
- task: Cache@2
  displayName: Cache task
  inputs:
    key: 'docker | "$(Agent.OS)" | "$(Build.SourceVersion)"'
    path: /tmp/.buildx-cache
    restoreKeys: 'docker | "$(Agent.OS)"'

- bash: |
    docker buildx create --driver docker-container --use
    docker buildx build --cache-to type=local,dest=/tmp/.buildx-cache-new --cache-from type=local,src=/tmp/.buildx-cache --push --target cloud --tag $REGISTRY_NAME/$IMAGE_NAME:$TAG_NAME .
  displayName: Build Docker image

# optional: set up deploy steps here

- task: Docker@2
  displayName: Logout of ACR
  inputs:
    command: logout
    containerRegistry: $(SERVICE_NAME)

The key here is to set up Docker buildx and run it with the --cache-to and --cache-from flags instead of using the Azure Docker task. You'll also need to use the Cache task to make sure the Docker cache is reloaded in subsequent pipeline runs, and you'll have to set up a manual swap step where the newly-generated cache replaces the old cache.

jidicula
  • 3,454
  • 1
  • 17
  • 38
  • 1
    In my case, I had to create the container from the image built using `buildx bake` commands and what @jidicula suggests is definitely right. However, I had to add the `--load` flag to load the image to the local store, since the `docker-container` driver creates a separate environment (i.e. you don't see the images using `docker images ls`): `docker buildx create --driver docker-container --use`; `docker buildx bake *--load*`; `docker create --name=containerName image:tag`; `docker cp containerName:// ./`; – Krusty Jul 19 '23 at 13:33
1

Had the same problem. Turns out it was the task "npm authenticate" that was breaking the caching by inserting a new token each time. I just put a static .npmrc file into the Pipeline > Library > SecureFiles depot and everything became unbelievably fast:

- task: DownloadSecureFile@1
  name: 'npmrc'
  displayName: 'Download of the npmrc authenticated'
  inputs:
    secureFile: '.npmrc'

- task: CopyFiles@2
  inputs:
    SourceFolder: $(Agent.TempDirectory)
    contents: ".npmrc"
    TargetFolder: $(Build.SourcesDirectory)/Code
    OverWrite: true
  displayName: 'Import of .npmrc'

- task: Docker@2
  displayName: Build an image
  inputs:
    command: build
    dockerfile: '$(Build.SourcesDirectory)/Dockerfile'
    tags: |
      $(tag)

The only drawback of this is that personnal access token last a year. So you need to replace your securefile every year...

Guillaume Blanchet
  • 461
  • 5
  • 11
1

You can use Docker@2 to pull the image and if you have a service connection to the ACR, there's no need run login command. Also, I didn't need to set the BUILDKIT_INLINE_CACHE, but if it's not working for you the following link explains how to set it and use it https://zws.im/‍. My solution is based on Kees Schollaart solution:

- task: Docker@2
  displayName: "Pull image"
  inputs:
    command: pull
    containerRegistry: "$(ContainerRegistryName)"
    arguments: $(ACR_ADDRESS)/$(REPOSITORY):latest
- task: Docker@2   
  displayName: build   
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    repository: '$(REPOSITORY)'
    command: 'build'
    Dockerfile: './dockerfile '
    buildContext: '$(BUILDCONTEXT)'
    arguments: '--cache-from=$(ACR_ADDRESS)/$(REPOSITORY):latest' 
    tags: |
      $(Build.BuildNumber)
      latest
- task: Docker@2   
  displayName: "push"   
  inputs:
    command: push
    containerRegistry: "$(ContainerRegistryName)"
    repository: $(REPOSITORY) 
    tags: |
      $(Build.BuildNumber)
      latest
jtvcodes
  • 11
  • 1