22

We are building CD pipeline using VSTS hosted build servers. It takes more than 3 minutes to restore Nuget. This is too much time.

How can I make it run faster? Is there any sort of caching system we can use?

Community
  • 1
  • 1
Rıfat Erdem Sahin
  • 1,738
  • 4
  • 29
  • 47
  • Please provide any context, project information or a [Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve) – Martin Ullrich Sep 01 '17 at 17:00

2 Answers2

6

UPDATE: Caching is now generally available (docs)

Caching is currently on the feature pipeline with a TBD date. In the mean time you can use the Upload Pipeline Artifact/Download Pipeline Artifact tasks to store results in your Azure DevOps account to speed up up/downloads.

The Work-in-progress can be tracked here.

In the mean time, the Microsoft 1ES (one engineering system, internal organization) has released their internal solution that uses Universal Packages to store arbitrary packages in your Azure DevOps account. It's very fast because it can sync the delta between previous packages. There is a sample on how to configure your Azure Pipeline to store the NuGet package cache in your Sources Directory in order for the task to cache them.

variables:
  NUGET_PACKAGES: $(Build.SourcesDirectory)/packages
  keyfile: '**/*.csproj, **/packages.config, salt.txt'
  vstsFeed: 'feed name'

steps:
- task: 1ESLighthouseEng.PipelineArtifactCaching.RestoreCache@1
  displayName: 'Restore artifact'
  inputs:
    keyfile: $(keyfile)
    targetfolder: $(NUGET_PACKAGES)
    vstsFeed: $(vstsFeed)
farlee2121
  • 2,959
  • 4
  • 29
  • 41
jessehouwing
  • 106,458
  • 22
  • 256
  • 341
  • 5
    Unfortunately the new cache is unbearably slow to restore, often slowing than just redownloading dependencies directly. – Adrian Baker Aug 03 '20 at 18:04
  • The problem is generally 45000 small files. There isn't really a fast way to cache that mess unless you can mount a drive image. Mount a vhd, restore the gazillion files, unmount the drive, cache the drive. – jessehouwing Aug 03 '20 at 18:31
  • 3
    That's not the problem, since it's still slow for a small set of larger files, and often slow even compared to CDN downloads outside the data center. Also, tools like Gradle Enterprise show that you can efficiently cache lots of small files. But yes, the cache should absolutely be local disk based, it was raised in the public comments on the design review that a network based cache would be slow enough to render it of little use, and sure enough, it is. I'm not sure what other CI/CD providers have attempted network-only based caching. – Adrian Baker Aug 04 '20 at 02:08
3

In my scenario, Nuget restore ran quickly when run interactively, but very slowly when run through CD pipeline (Jenkins). Setting revocation check mode to offline reduced my Nuget restore times from 13+ minutes to under 30 seconds (I found this solution here)

I set an environment variable in my build script prior to running Nuget restore:

  • SET NUGET_CERT_REVOCATION_MODE=offline

Disclaimer: Turning off certificate revocation has implications - see this link.

Grits
  • 121
  • 1
  • 5