5

Background: Pipeline caching allows one to store a folder and re-use it on the next build, given that some keys, branches and whatnot matches. Useful for node_modules, nuget packages and Git LFS, but also for builds

I can't find anything in the Pipeline Caching docs for this.

I want to save the cache even if the build fails, for incremental builds or Git LFS checkouts, such as the following scenario:

  • new feature branch checked in
  • a big refactor, thus many changes
  • a few tests fail, so cache is not stored
  • fixing them are trivial, and require small recompile, but the entire build needs to be re-run the entire pipeline failed.

The example list something like this:

- task: Cache@2
  inputs:
    key: 'yarn | "$(Agent.OS)" | yarn.lock'
    restoreKeys: |
       yarn | "$(Agent.OS)"
       yarn
    path: $(YARN_CACHE_FOLDER)
  displayName: Cache Yarn packages

But it only ever caches if the entire pipeline succeeds.

Macke
  • 24,812
  • 7
  • 82
  • 118
  • 2
    What do you mean by "pipeline cache"? You're using your own terminology, no one else is going to understand what you mean. Please explain the specific problem you're having and what your desired goal is. – Daniel Mann Jun 25 '20 at 15:48
  • The function you describe seems abstract, are there any similar examples? Do you want to build incrementally? – Leo Liu Jun 26 '20 at 08:35
  • 2
    @DanielMann I'm refering to Auzure Devops Pipeline Caching, see link. It's an official term. – Macke Jun 26 '20 at 13:43
  • @LeoLiu-MSFT Yes. The specific problem is that I want to cache my build folder to do incremental building, but it won't save the cached folder if the build fails. – Macke Jun 26 '20 at 13:43

2 Answers2

6

How do i store Azure DevOps pipeline cache even if a step fail (like tests)?

After a period of investigation and discussion, I am afraid there is no such task/feature to store Azure DevOps pipeline cache even if a step fail.

According to the document Pipeline caching, this task are used to help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost to recreate or redownload the same files again.

It does not parse the file or the build log. So, even if we cached the file generated by the failed pipeline, we cannot use that file that was not generated correctly or not generated during the next time. It seems very difficult to extract the correct information in a failed build.

Hope this helps.

Leo Liu
  • 71,098
  • 10
  • 114
  • 135
  • 2
    Hi Leo. Thanks for spending time to looking into it. I'd of course appreciate if the decision on what to do on failed builds was up to me, and not the system. If the build succeeds, but not the tests, I can cache the build? If the nuget package download succeeds, but not the build, I could still cache the packages, etc. I'll mark this as the accepted answer and hope, but not expect, that the future might have a fix for this. :) – Macke Jul 01 '20 at 06:46
  • 2
    @Macke, Yes, I agree with you. It seems we need something like `task caching` instead of `pipeline caching`, so that we could cache the output for the task. I think this is a good idea, but it may take a long time to realize it. – Leo Liu Jul 01 '20 at 06:51
  • 3
    Well, as you define the folder for pipeline caching, I've mostly used it as task caching anyway, with different keys and dirs. Maybe I'll submit a PR on the cache task just to make the success check optional .. :) – Macke Jul 01 '20 at 07:54
1

I had the same problem with a code quality tool which has a valid cache to store, whether it fails or succeeds.

I fixed it with 2 jobs.

The first one is your original job with the cache task and you add steps at the end to publish your cache as a pipeline artifact. These additional steps should have condition: failed() so that you do this only if the cache task won't store its result.

The second job depends on the first one with condition: failed(). It also has the cache task, and you download your artifact to it so that the cache task this time stores the result.

Christian Fecteau
  • 11
  • 1