9

How can I programmatically get all dependencies of a Maven module outside a Maven execution environment?

So far I have:

via maven-core:

Path pomPath = ...;
MavenXpp3Reader reader = new MavenXpp3Reader();
try (InputStream is = Files.newInputStream(pomPath)) {
    Model model = reader.read(is);
    this.mavenProject = new MavenProject(model);
}

and via jcabi-aether:

File localRepo = Paths.get(System.getProperty("user.home"), ".m2").toFile();
new Classpath(mavenProject, localRepo, "runtime")

Is this generally correct so far?

The issue now is that I'm getting a NullPointerException:

Caused by: java.lang.NullPointerException
    at com.jcabi.aether.Aether.mrepos(Aether.java:197)
    at com.jcabi.aether.Aether.<init>(Aether.java:140)
    at com.jcabi.aether.Classpath.<init>(Classpath.java:125)

since mavenProject.getRemoteProjectRepositories() returns null.

How can I initialze the MavenProject to contain the configured remote repos taking the settings.xml file (mirrors, proxies, repositories etc.) into account as well?

Tunaki
  • 132,869
  • 46
  • 340
  • 423
Puce
  • 37,247
  • 13
  • 80
  • 152
  • I don't think this is going in the right direction. Take a look at [`dependency:get`](http://maven.apache.org/plugins/maven-dependency-plugin/get-mojo.html) instead – janos Nov 25 '16 at 22:44
  • If you don't have the POM file, and only have coordinates, you can look into Aether. To fetch direct dependencies with Aether, [you can refer to this answer of mine](http://stackoverflow.com/questions/39638138/find-all-direct-dependencies-of-an-artifact-on-maven-central). If you do have the POM, then using `dependency:get -Dtransitive=false` will get the direct dependencies in a much easier way. Take a look at the [Invoker API](http://maven.apache.org/shared/maven-invoker/) – Tunaki Nov 25 '16 at 22:48
  • To clarify: I do have a POM file but the code runs outside a Maven execution. I want to populate a URLClassLoader with the URLs to the dependencies in the local repo (which might have to be downloaded first from a remote repo). I don't see how a plugin is going to help me... – Puce Nov 25 '16 at 22:54
  • @Tunaki thanks, I'll check your sample – Puce Nov 25 '16 at 23:08
  • @Tunaki using your sample: how can I take the settings.xml file (mirrors, proxies, repositories etc.) into account and how can I get all transitive dependencies for the "runtime" scope? – Puce Nov 25 '16 at 23:29
  • Aether doesn't provide a way to read the Maven settings file, so you can either use the existing methods like `setMirrorSelector` and `setProxySelector` on the system session. But if you really want to read the Maven settings, using the `maven-settings-builder` library would be easier, [see also the sample code mentioned here](https://wiki.eclipse.org/Aether/Creating_a_Repository_System_Session). – Tunaki Nov 25 '16 at 23:40
  • @Puce I made an edit regarding proxy and mirrors with Aether on the linked answer. Btw, what do you want to do with the dependencies after getting them? – Tunaki Nov 26 '16 at 00:17
  • @Tunaki Thanks! I need to dynamically configure a URLClassLoader which can resolve classes and resources from all dependencies defined by a POM file. Is there a way to get all transitive dependencies with "runtime" scope using Aether? The samples I found so far ony resolve all direct dependencies and I think e.g. "test" scope is included as well. – Puce Nov 26 '16 at 00:25
  • @Puce Yep, take a look at [`system.resolveDependencies`](http://download.eclipse.org/aether/aether-core/0.9.0/apidocs/org/eclipse/aether/RepositorySystem.html#resolveDependencies(org.eclipse.aether.RepositorySystemSession,%20org.eclipse.aether.resolution.DependencyRequest)). It takes a `DependencyRequest` as argument, on which you can set specific `DependencyFilter`, like [`ScopeDependencyFilter`](http://download.eclipse.org/aether/aether-core/0.9.0/apidocs/org/eclipse/aether/util/filter/ScopeDependencyFilter.html), which would retain only the dependencies with the given scope. – Tunaki Nov 26 '16 at 00:36
  • (Continued) But since the linked question was about direct dependencies, I will answer this one (tomorow, it's late now :) ) with how to fetch transitive dependencies with the filters. Could you edit with all of your requirements? – Tunaki Nov 26 '16 at 00:37
  • [This approach](https://stackoverflow.com/questions/13200497/how-to-programmatically-list-all-transitive-dependencies-including-overridden-o#70823705) is for use within a plugin, but may be usable outside as well if the `LifecycleDependencyResolver` can be properly instantiated; it doesn't have a direct dependency on Aether (although Maven would obviously use Aether under the hood) – Janaka Bandara Jan 23 '22 at 16:16

1 Answers1

14

Outside of a Maven plugin, the way to operate on artifacts is through Aether. The team has a sample project to get the transitive dependencies of a given artifact called ResolveTransitiveDependencies. Once you have the Aether dependencies set up (like shown here), you can simply have:

public static void main(final String[] args) throws Exception {
    DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator();
    RepositorySystem system = newRepositorySystem(locator);
    RepositorySystemSession session = newSession(system);

    RemoteRepository central = new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2/").build();

    Artifact artifact = new DefaultArtifact("group.id:artifact.id:version");

    CollectRequest collectRequest = new CollectRequest(new Dependency(artifact, JavaScopes.COMPILE), Arrays.asList(central));
    DependencyFilter filter = DependencyFilterUtils.classpathFilter(JavaScopes.COMPILE);
    DependencyRequest request = new DependencyRequest(collectRequest, filter);
    DependencyResult result = system.resolveDependencies(session, request);

    for (ArtifactResult artifactResult : result.getArtifactResults()) {
        System.out.println(artifactResult.getArtifact().getFile());
    }
}

private static RepositorySystem newRepositorySystem(DefaultServiceLocator locator) {
    locator.addService(RepositoryConnectorFactory.class, BasicRepositoryConnectorFactory.class);
    locator.addService(TransporterFactory.class, FileTransporterFactory.class);
    locator.addService(TransporterFactory.class, HttpTransporterFactory.class);
    return locator.getService(RepositorySystem.class);
}

private static RepositorySystemSession newSession(RepositorySystem system) {
    DefaultRepositorySystemSession session = MavenRepositorySystemUtils.newSession();
    LocalRepository localRepo = new LocalRepository("target/local-repo");
    session.setLocalRepositoryManager(system.newLocalRepositoryManager(session, localRepo));
    return session;
}

It will download the artifacts and place them into "target/local-repo".

Note that you can configure proxy and mirrors with the DefaultProxySelector and DefaultMirrorSelector on the system session. It would be possible to read the Maven settings file and use it to populate the session, but things get really ugly really fast...


When you want tight coupling with Maven itself because you have access to the POM to process and want to take the settings into account, it is a lot simpler to directly invoke Maven programmatically. In this case, you're interested in the path of each dependencies, including transitive dependencies, of a given POM file. For that the dependency:list goal, together with setting the outputAbsoluteArtifactFilename to true, will give (almost) exactly that.

To invoke Maven programmatically, it is possible to use the Invoker API. Adding the dependency to your project:

<dependency>
  <groupId>org.apache.maven.shared</groupId>
  <artifactId>maven-invoker</artifactId>
  <version>2.2</version>
</dependency>

you can have:

InvocationRequest request = new DefaultInvocationRequest();
request.setPomFile(new File(pomPath));
request.setGoals(Arrays.asList("dependency:list"));
Properties properties = new Properties();
properties.setProperty("outputFile", "dependencies.txt"); // redirect output to a file
properties.setProperty("outputAbsoluteArtifactFilename", "true"); // with paths
properties.setProperty("includeScope", "runtime"); // only runtime (scope compile + runtime)
// if only interested in scope runtime, you may replace with excludeScope = compile
request.setProperties(properties);

Invoker invoker = new DefaultInvoker();
// the Maven home can be omitted if the "maven.home" system property is set
invoker.setMavenHome(new File("/path/to/maven/home"));
invoker.setOutputHandler(null); // not interested in Maven output itself
InvocationResult result = invoker.execute(request);
if (result.getExitCode() != 0) {
    throw new IllegalStateException("Build failed.");
}

Pattern pattern = Pattern.compile("(?:compile|runtime):(.*)");
try (BufferedReader reader = Files.newBufferedReader(Paths.get("dependencies.txt"))) {
    while (!"The following files have been resolved:".equals(reader.readLine()));
    String line;
    while ((line = reader.readLine()) != null && !line.isEmpty()) {
        Matcher matcher = pattern.matcher(line);
        if (matcher.find()) {
            // group 1 contains the path to the file
            System.out.println(matcher.group(1));
        }
    }
}

This creates an invocation request which contains: the goals to invoke and the system properties, just like you would launch mvn dependency:list -Dprop=value on the command-line. The path to the settings to use will default to the standard location of "${user.home}/settings.xml", but it would also be possible to specify the path to the settings with request.setUserSettingsFile(...) and request.setGlobalSettingsFile(...). The invoker needs to be set the Maven home (i.e. installation directory), but only if the "maven.home" system property isn't set.

The result of invoking dependency:list is redirected to a file, that is later post-processed. The output of that goal consists of the list of dependencies in the format (the classifier may be absent, if there are none):

group.id:artifact.id:type[:classifier]:version:scope:pathname

There isn't a way to output only the path of the resolved artifact's file, and the fact that the classifier may be absent complicates the parsing a bit (we can't split on : with a limit, since the path could contain a :...). First, the resolved artifacts are below the line "The following files have been resolved:" in the output file, then, since the wanted scope are only compile or runtime, we can get the path of the artifact's file with a simple regular expression that takes everything which is after compile: or runtime:. That path can then directly be used as a new File.

If the hoops during post-processing look too fragile, I guess you could create your own plugin that just outputs the resolved artifact's filename.

Community
  • 1
  • 1
Tunaki
  • 132,869
  • 46
  • 340
  • 423
  • Thanks, this helped me to implement a first version with settings support. – Puce Jan 21 '17 at 20:06
  • Only note for those who are coming here and have one master _pom.xml_ file with modules it is useful to set `appendOutput` flag on `true` to have a list of all dependencies from all modules: `properties.setProperty("appendOutput", "true");` Otherwise each module will overwrite output from the previous one. – cgrim Sep 10 '18 at 10:32
  • @Tunaki Can we achieve this use-case without having maven in the computer? – Pramodya Mendis Mar 19 '20 at 09:48
  • @Puce I'm trying to use the aether example. but I'm not able to resolve dependencies. I get exception `org.eclipse.aether.resolution.DependencyResolutionException: Failed to read artifact descriptor for ch.qos.logback:logback-classic:jar:1.2.3`. I only changed the follwoing line in the above code. `Artifact artifact = new DefaultArtifact("ch.qos.logback:logback-classic:1.2.3");` – Pramodya Mendis Mar 19 '20 at 10:57
  • Is there a working solution that uses Aether `4.0.0-alpha-2`? – Cardinal System Oct 26 '22 at 13:12