How to cache maven repository between builds?

Question

My goal is to be able to build, package and test a java project that is built with maven using a councourse build pipeline.

The setup as such is in place, and everything runs fine, but build times are much too long due to poor maven download rates from our nexus.

My build job yml file uses the following resource as base for the maven build:

# ...
image_resource:
  type: docker-image
  source:
    repository: maven
    tag: '3.3-jdk-8'
# ...

I am aware of the fact that having a "blank slate" for every buils is somwhat built into concourse by design.

Now my question is: what would be a good way to cache a local maven repository (say, with at least some basic stuff inside like Spring and it's dependencies)?

Following options come to my mind:

1. Using a docker image that has the dependencies built-in already
2. Creating a ressource that provides me the needed dependencies

As far as I can see, option 1) will not make the download size for the build smaller, as concourse seems to not cache docker images used as base for the build jobs (or am I wrong here?)

Before I move on, I would like to make sure that following option 2) gives me any advantage - does concourse cache docker images used as ressources?

I might miss out something, as I am relatively new to councourse. So forgive me if I force you to state the obvious here. :)

Answer 1

• Assuming that your Nexus is local, I would look into why there are poor download rates from that, as using something like Nexus and Artifactory locally is currently the easiest way to do caching. They will manage the lifetime of your cached dependencies, so that you don't have dependencies being cached longer that they are needed and new dependencies are adding as they are used.
• If you want to share a cache between tasks of a job, then output the cached dependencies folder (.m2 folder for maven) of a task and use that as an input of another task. For reference, see following example:

 ---
 jobs:
   - name: create-and-consume
     public: true
     plan:
       - task: make-a-file
         config:
           platform: linux
           run:
             # ...
           outputs:
             # ensure that relative .m2 folder is used: https://stackoverflow.com/a/16649787/5088458
             - name: .m2
       - task: consume-the-file
         config:
           platform: linux
           inputs:
             - name: .m2
           run:
             # ...

• If you want to share a cache between all executions of a single task in a job, tasks can also be configured to cache folders.
• If you want to cached between jobs then you could:
  • build a docker image with the cached folder, but then you'll need to manage that when dependencies are updated, although that may be possible via another pipeline.
  • create a resource that manages the cache for you. For example look at gradle-cache-resource or npm-cache-resource, although they require that the input is from git-resource.

I think Concourse CI does cache docker images used for tasks, but can also have them as resources of your pipeline and then use the image parameter of the task to pass that resource. You can see what is cached and for how long using the volumes command of fly.

Answer 2

I can confirm that concourse does in fact cache docker images, so creating an image that has all the dependencies might not be a bad option.

Answer 3

You could use the same mechanism the git resources type use for caching for your maven cache, just as they do for now here: https://github.com/ymedlop-sandbox/npm-cache-resource (Not my repo)