Find unresolved maven dependencies in multi-module maven project

Question

I'd like maven to report unresolved dependencies in multi-module maven Java project which has below structure:

multi-module-java-app/
├── app1
│   ├── pom.xml
│   └── src
├── app2
│   ├── pom.xml
│   └── src
└── pom.xml

poms are at the bottom.

Background:

Maven is used as a dependency management and build tool. Artifactory is repository manager. Artifacts may be built using maven locally on a developer’s environment or on Jenkins build server. Artifactory periodically moves artifacts to special archive repository which is part of all repository virtual.

Maven caches locally built artifacts in~/.m2 directory on the computer where Maven runs whether it’s developer environment or build server.

Problem

Several issues may arise:

Local builds on developers’ VMs may succeed, but fail in Jenkins.

Local builds, Jenkins builds may succeed, but fail on another developer’s VM.

Cause

Present/missing artifacts on a developer .m2 cache, missing artifacts in build server’s .m2 cache and/or archive Artifactory repository

Proposed solution

Run [path_to_maven]mvn dependency:list. [path_to_maven] has custom maven installation (with empty .m2 cache) in the root folder of the project. Custom maven is also configured using settings.xml to use special non-archived repository (all repository without the archive). The output is like below:

It reports unresolved dependencies as well as dependent artifacts which miss them. However this solution has 2 main drawbacks:

.m2 missing slows down the detection significantly as all the dependencies have to be downloaded
unresolved artifacts or modules which miss them are always snapshots.

Running mvn -B -q versions:set -DnewVersion=[some_version] solves the second. This command runs during release pipeline anyway.

However it's not clear how to solve the first.

How to find unresolved maven dependencies while using .m2, so that unresolved dependencies may be detected quickly during Jenkins build after each push to feature branch?

The only idea that pops is that .m2 on build server will be synced with Artifactory.

.m2 on developers' machines can be synced as well using some sort of custom plugin that uses rsync. Is there a known plugin that does this?

The ultimate goal is to remove archive repository and let the builds fail. But, first developers need to align the dependencies to the latest versions.

root pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
 
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
 
  <groupId>com.mycompany.app</groupId>
  <artifactId>multi-module-java-app</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
 
  <name>multi-module-java-app</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>
 
  <pluginRepositories>    
      <pluginRepository>
          <id>plugins</id>
          <name>plugins</name>
          <url>http://localhost:8081/artifactory/all</url>
      </pluginRepository>
  </pluginRepositories>
 
  <repositories>
      <repository>
          <id>all</id>
          <name>all</name>
          <url>http://localhost:8081/artifactory/all</url>
      </repository>
  </repositories>
 
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.example</groupId>
        <artifactId>spring-boot</artifactId>        
      </dependency>
    </dependencies>
  </dependencyManagement>
 
  <modules>
      <module>app1</module>
      <module>app2</module>
  </modules>
 
</project>

app 1 pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
 
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
        <groupId>com.mycompany.app</groupId>
        <artifactId>multi-module-java-app</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
  </parent>
  <groupId>com.mycompany.app.app1</groupId>
  <artifactId>app1</artifactId>
  <version>1.0-SNAPSHOT</version>
 
  <name>app1</name>
 
  <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <finalName>${artifactId}</finalName>
                </configuration>
            </plugin>
        </plugins>
  </build>
  <pluginRepositories>    
      <pluginRepository>
          <id>plugins</id>
          <name>plugins</name>
          <url>http://localhost:8081/artifactory/all</url>
      </pluginRepository>
  </pluginRepositories>
 
  <repositories>
      <repository>
          <id>all</id>
          <name>all</name>
          <url>http://localhost:8081/artifactory/all</url>
      </repository>
  </repositories>
 
  <dependencies>
    <dependency>
      <groupId>com.example</groupId>
      <artifactId>spring-boot</artifactId>
      <version>0.0.1-20200510.095344-1</version>
    </dependency>
  </dependencies>
 
</project>

app2 pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
 
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
        <groupId>com.mycompany.app</groupId>
        <artifactId>multi-module-java-app</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
  </parent>
  <groupId>com.mycompany.app.app2</groupId>
  <artifactId>app2</artifactId>
  <version>1.0-SNAPSHOT</version>
 
  <name>app2</name>
  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-jar-plugin</artifactId>
              <configuration>
                  <finalName>${artifactId}</finalName>
              </configuration>
          </plugin>
      </plugins>
  </build>
 
  <pluginRepositories>    
      <pluginRepository>
          <id>plugins</id>
          <name>plugins</name>
          <url>http://localhost:8081/artifactory/all</url>
      </pluginRepository>
  </pluginRepositories>
 
  <repositories>
      <repository>
          <id>all</id>
          <name>all</name>
          <url>http://localhost:8081/artifactory/all</url>
      </repository>
  </repositories>
 
  <dependencies>
    <dependency>
      <groupId>com.example</groupId>
      <artifactId>spring-boot</artifactId>
      <version>0.0.1-20200510.095344-1</version>
    </dependency>
  </dependencies>
 
</project>

Wow, that's a scholastic context. Part of problem 1 is, it's not a problem at all. That's how maven works; you start w/empty local pom and it resolves what you list as dependencies ( and tne transitive dependencies). What you want is `dependency:analyze`. That aligns your pom with your imports. Clean pom is the way to go. You can also keep module versions aligned by specifying GAV dependencies in root pom but only GA dependencies in sub-modules. — Ian W, Jul 15 '20 at 10:33
I'm not sure why you have this discrepancy between developers repositories and artifactory. Are developers building and installing dependencies locally which haven't been released? — tgdavies, Jul 15 '20 at 10:36
@tgdavies yes, parent and modules have SNAPSHOT versions in pom.xml and developers need to verify that local builds pass before pushing to feature branch after which build will be triggered in Jenkins, which should fail if unresolved dependencies are found — rok, Jul 15 '20 at 10:41
I have not understood yet why you could not just wait till Jenkins builds. If dependencies are missing, the Jenkins build fails and the developer can repair the problem. — J Fabian Meier, Jul 15 '20 at 11:17
I don't understand which dependencies are present locally but missing in artifactory. I understand that app1 and app2 will have SNAPSHOT versions, but that's fine as they are all built together. — tgdavies, Jul 15 '20 at 12:33
I understand you have archived artifacts that might be missing as dependencies. But why not just try a build with Jenkins and if it fails, look at the error message? Why do you need some process to figure out before the build if artifacts are missing? — J Fabian Meier, Jul 15 '20 at 14:18
@JF Meier. It's because of the .m2 cache which present on Jenkins build server and developers' machines. It may have the archived artifact, but Jenkins .m2 may not and vice versa. .m2 cache differences may exist even between different developer machines — rok, Jul 15 '20 at 14:26
Yes. But I still don't see the big issue: Either Jenkins builds successfully (everything is fine) or Jenkins fails. In the second case the developer sees in the log that some artifact was probably archived and repairs this. — J Fabian Meier, Jul 15 '20 at 14:36
And if suspect a problem due to an archived artifact, you can run https://maven.apache.org/plugins/maven-dependency-plugin/purge-local-repository-mojo.html and find out. — J Fabian Meier, Jul 15 '20 at 14:37
@JF Meier. The ultimate goal is to remove `archive` repository. But, first, developers should use the latest versions of dependencies. That's why, automatic process of resolving dependencies from non-archived repo and without .m2 cache was needed. — rok, Jul 15 '20 at 14:46
@JF Meier. Thanks. This plugin seems helpful in Jenkins and local builds. But it will slow down builds significantly as all the dependencies would need to be re-downloaded. — rok, Jul 15 '20 at 14:48

score 4 · Answer 1 · answered Jul 19 '20 at 17:59

I have run into this exact issue where the developer's local build environment was different from Jenkins Slave environment. In the ideal world, the developer needs to baseline their local environment with that of the slave, or depend entirely on Jenkins job builds once initial stages of development are complete.

I appreciate the fact that you are trying to provide an automated sync feature of the .m2 repositories, it is feasible but adds scope for error and additional routine maintenance tasks, not to mention user education issues. For instance, how will you ensure that the .m2 is the latest version? In terms of maven dependencies, the developer knows best, or they may be introducing new dependencies which do not exist on the slave yet. Thereby, I suggest fixing the root problem of developer not aligning their dependencies which is more design related than automation.

Not sure if you want to take this path, but may help someone:

Eliminate the need for .m2 repositories in local machines of developers. The m2 cache creates a problem, if the developer's machine gets wiped out or corrupt and there will be a need for updates, audits and reconciliation.
Eliminate the need for .m2 in Jenkins slaves. The problem here is, multiple slaves tend to have different .m2 cache content and syncing back and forth from Artifactory and then developers also syncing it to their local sounds complicated. There is no saying that all these .m2 will be in sync at any point of time and a builds may still get executed with an n-1 version of it.
Now that there is no .m2 , we still need a place for developers to pull dependencies from. Push all your dependencies to a repo in Artifactory and actively maintain it. Use the setting.xml feature to pull the standard dependencies. If a developer is building in his local machine, via Eclipse or other, the dependencies will be available to be pulled from the IDE itself by using the same xml reference so there is no local cache maintained on the developer's machine.
When the build environment has a minor difference between the local and the Jenkins Slave This causes the .jar to be the same size or slightly different in kilobytes of size.

To identify this difference between jar files use any tools listed here in this post, it also helps developers to identify which dependency is out of sync, by themselves: Comparing two .jar files

If this design is implemented, the Artifactory repository containing the dependencies becomes the single source of truth for dependencies, you need to work with developers to create a cadence as to how the dependencies will be updated and consumed to/from this single source of truth. I hope this is useful.

this useful, thanks. indeed, first will align dependencies and then will see. but i still , builds whether local or remote will be terribly slow without .m2... that's why i ask for solution with .m2. — rok, Jul 25 '20 at 19:10
Thanks! Are you saying the build duration will be slow or network latencies cause builds to be slow i.e., firewalls between `jenkins` and `artifactory`? Are they all on the same network? — mdabdullah, Jul 25 '20 at 20:57
local build duration will be slow because all dependencies would have to be redownloaded on each build. Regarding jenkins builds, will need to ensure `jenkins` and `artifactory` are on the same network — rok, Jul 26 '20 at 06:09
Bring them on the same network, solving that network problem may be easier. The other thing is I am almost sure `Artifactory` does not do a complete checkout each time, only the `delta` is checked out based on a `checksum` value ? This will be something you can check with JFrog too. — mdabdullah, Jul 26 '20 at 06:16

Find unresolved maven dependencies in multi-module maven project

1 Answers1