fetch all maven dependencies including plugin dependencies

Question

I'm trying to get maven to download all dependencies (compile, test, plugins, etc.) so that I can avoid having our dockerized builds waste unnecessary time downloading them over and over again.

We have dockerized our maven build so that we can run it from our jenkins without having a lot of build specific dependencies installed on the jenkins machine (Java, redis, maven dependencies, etc.). Our build relies on incremental docker builds that only executes steps that actually need re-running.

Our main build is a DockerFile that has several steps to install the jdk, maven, etc. Then it does a

COPY ./pom.xml /opt/inbot-api/pom.xml RUN mvn dependency:copy-dependencies clean

This will download dependencies to the local maven repository and then cleans out the target directory.

Then we copy the source tree to the image and run the full build.

COPY ./src /opt/inbot-api/src RUN mvn -e clean install

The general idea is that on a clean machine, docker will execute all the RUN steps but on incremental builds it will only rerun things that need re-running. After each run step, it stores an intermediary image. So, if the pom file doesn't change, there's no need to rerun the dependency fetching step because it would produce the exact same outcome. So, instead it loads the cached intermediary image with all the dependencies already downloaded. This is exactly what we want.

There's a lot more to our DockerFile that is not so relevant here but ultimately it produces a docker file with our compiled artifacts, an nginx config and all our runtime dependencies that we can deploy to ECS.

This nearly works except the mvn clean install still downloads additional plugin dependencies on every build. So, these are dependencies that the copy-dependencies step does not cover.

My question, how do I get RUN mvn dependency:copy-dependencies clean to download all dependencies including the plugin dependencies. I've seen people actually do a mvn verify clean instead of mvn dependency:copy-dependencies clean but that is kind of slow in our case. I was wondering if there was a better way to do this.

I'd appreciate any feedback on how to improve this.

Update

I now do a

RUN mvn -B -T 4 dependency:copy-dependencies dependency:resolve-plugins dependency:go-offline clean

And it still downloads more stuff with the mvn clean install after that. A mvn -o clean install still fails, despite the dependency:go-offline. So, it seems this plugin is broken.

For the plugin i would suggest to use `mvn dependency:resolve-plugins` see https://maven.apache.org/plugins/maven-dependency-plugin/resolve-plugins-mojo.html — khmarbaise, Dec 10 '15 at 13:32
Please look at my solution at https://stackoverflow.com/a/71066133/418599 . — Antonio Petricca, Feb 10 '22 at 14:17
Thanks but I've long stopped using maven; I'm now using Gradle. Which of course has its own set of challenges regarding offline usage. — Jilles van Gurp, Feb 12 '22 at 19:31

score 3 · Answer 1 · answered Dec 11 '18 at 20:51

3

This works for me, no other dependencies to download:

RUN mvn -B dependency:resolve dependency:resolve-plugins

answered Dec 11 '18 at 20:51

Pavel Sklenar

51
2

Also bear in mind that this command will only resolve the dependencies and plugins in active profiles, so if you've got a profile that you want to execute in the later no-downloads build, you'll _also_ have to activate it in the dependency resolution build. – ryanm Aug 03 '22 at 07:50

score 1 · Answer 2 · answered Dec 10 '15 at 16:20

1

For the plugin i would suggest to use mvn dependency:resolve-plugins

See the documentation: https://maven.apache.org/plugins/maven-dependency-plugin/

answered Dec 10 '15 at 16:20

khmarbaise

92,914
28
189
235

1

this nearly works but it it still downloads some other stuff after I do a `mvn dependency:copy-dependencies dependency:resolve-plugins clean`. I tried running the second mvn invocation with -o and then it fails: complaining about these dependencies: "The following artifacts could not be resolved: org.apache.maven.reporting:maven-reporting-api:jar:2.0.9, org.apache.maven.doxia:doxia-sink-api:jar:1.0-alpha-10" this seems to be used by the maven enforcer plugin indirectly, which we use indeed. – Jilles van Gurp Dec 10 '15 at 19:41
The 2nd invocation is running additional downloads because you are calling the `install` builtin target for the first time. If you add the `install` target to your mvn first invocation (before you add the source) you'll solve your problem. – Trastle Dec 26 '17 at 15:23
1

I'm trying to avoid doing the full build here. The whole point of this is to get docker to cache dependencies in a separate layer before it does the build and before I copy over the source files. In any case, I've moved on to gradle. Same kind of issues there. I don't think this can be done cleanly with either maven or gradle for most non trivial projects. Would be nice if it could. But it seems there are always a few plugins that insist on runtime downloads of whatever. – Jilles van Gurp Jul 26 '18 at 12:31

score -1 · Answer 3 · answered Jul 25 '18 at 13:19

-1

I almost resolve with this:

RUN mvn install -DskipTests dependency:resolve dependency:resolve-plugins

answered Jul 25 '18 at 13:19

DET66

183
2
2
8

1

What do you mean by: *almost resolved*? Did this solve the problem or not? – Noel Widmer Jul 25 '18 at 13:36
2

In my project i have `maven-surefire-plugin` plugin who have own dependencies, what i did to fix that, after i run above command i run a 'mvn test' and see what dependecies are downloaded, and i put those dependecies in `` and was ok. This is a WORKAROUND, but it's working for me. – DET66 Jul 27 '18 at 11:16

fetch all maven dependencies including plugin dependencies

3 Answers3