24

I can't seem to find any info on whether scanning all available classes (for interfaces, annotations etc) is still possible in runtime, the way Spring, Reflections and many other frameworks and libraries currently do, in the face of Jigsaw related changes to the way classes are loaded.

EDIT: This question is about scanning the real physical file paths looking for classes. The other question is about dynamically loading classes and resources. It's related but very much not a duplicate.

UPDATE: Jetty project has made a JEP proposal for a standardized API for this. If you have a way to help make this reality, please do. Otherwise, wait and hope.

UPDATE 2: Found this relevant sounding post. Quoting the code snippet for posterity:

If you are really just looking to get at the contents of the modules in the boot layer (the modules that are resolved at startup) then you'll do something like this:

  ModuleLayer.boot().configuration().modules().stream()
         .map(ResolvedModule::reference)
         .forEach(mref -> {
             System.out.println(mref.descriptor().name());
             try (ModuleReader reader = mref.open()) {
                 reader.list().forEach(System.out::println);
            } catch (IOException ioe) {
                 throw new UncheckedIOException(ioe);
             }
         });
kaqqao
  • 12,984
  • 10
  • 64
  • 118
  • I don't think much has changed, I mean if there is visibility of one module towards another, you can still do the usual things. In case you can't access a class from a module you would probably hit an exception like `InaccessibleObjectException` or something like that – Eugene Jan 30 '17 at 11:01
  • @Eugene You sure? Asking because classpath now seems to be a "legacy" thing, getting replaced by _modulepath_. So I'm guessing scanning what used to be classpath has changed... – kaqqao Jan 30 '17 at 11:18
  • Possible duplicate of [Loading classes and resources in Java 9](https://stackoverflow.com/questions/45166757/loading-classes-and-resources-in-java-9) – Michael Easter Aug 01 '17 at 04:58
  • @MichaelEaster It is related, but not a duplicate. This one is about scanning directories and URLs for available classes, that one is about loading them. – kaqqao Aug 01 '17 at 07:23
  • 1
    See my answer below for scanning more than just the boot module layer. – Luke Hutchison Jun 28 '18 at 00:25
  • 1
    https://github.com/classgraph/classgraph/wiki/Code-examples The ClassGraph (previously FastClassGraph) has been helpful for me. – phreed Jul 25 '19 at 13:47

2 Answers2

30

The following code achieves module path scanning in Java 9+ (Jigsaw / JPMS). It finds all classes on the callstack, then for each class reference, calls classRef.getModule().getLayer().getConfiguration().modules(), which returns a a List<ResolvedModule>, rather than just a List<Module>. (ResolvedModule gives you access to the module resources, whereas Module does not.) Given a ResolvedModule reference for each module, you can call the .reference() method to get the ModuleReference for a module. ModuleReference#open() gives you a ModuleReader, which allows you to list the resources in a module, using ModuleReader#list(), or to open a resource using Optional<InputStream> ModuleReader#open(resourcePath) or Optional<ByteBuffer> ModuleReader#read(resourcePath). You then close the ModuleReader when you're done with the module. This is not documented anywhere that I have seen. It was very difficult to figure all this out. But here is the code, in the hope that someone else will benefit from this.

Note that even in JDK9+, you can still utilize traditional classpath elements along with module path elements, so for a complete module path + classpath scan, you should probably use a proper classpath scanning solution, such as ClassGraph, which supports module scanning using the below mechanism (disclaimer, I am the author). You can find a reflection-based version of the following code here.

Also note that there was a bug in StackWalker in several JDK releases after JDK 9 that has to be worked around, see the above reflection-based code for details.

package main;

import java.lang.StackWalker;
import java.lang.StackWalker.Option;
import java.lang.StackWalker.StackFrame;
import java.lang.module.ModuleReader;
import java.lang.module.ModuleReference;
import java.lang.module.ResolvedModule;
import java.net.URI;
import java.security.AccessController;
import java.security.PrivilegedAction;
import java.util.AbstractMap.SimpleEntry;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Deque;
import java.util.HashSet;
import java.util.List;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Stream;

public class Java9Scanner {

    /** Recursively find the topological sort order of ancestral layers. */
    private static void findLayerOrder(ModuleLayer layer,
            Set<ModuleLayer> visited, Deque<ModuleLayer> layersOut) {
        if (visited.add(layer)) {
            List<ModuleLayer> parents = layer.parents();
            for (int i = 0; i < parents.size(); i++) {
                findLayerOrder(parents.get(i), visited, layersOut);
            }
            layersOut.push(layer);
        }
    }

    /** Get ModuleReferences from a Class reference. */
    private static List<Entry<ModuleReference, ModuleLayer>> findModuleRefs(
            Class<?>[] callStack) {
        Deque<ModuleLayer> layerOrder = new ArrayDeque<>();
        Set<ModuleLayer> visited = new HashSet<>();
        for (int i = 0; i < callStack.length; i++) {
            ModuleLayer layer = callStack[i].getModule().getLayer();
            findLayerOrder(layer, visited, layerOrder);
        }
        Set<ModuleReference> addedModules = new HashSet<>();
        List<Entry<ModuleReference, ModuleLayer>> moduleRefs = new ArrayList<>();
        for (ModuleLayer layer : layerOrder) {
            Set<ResolvedModule> modulesInLayerSet = layer.configuration()
                    .modules();
            final List<Entry<ModuleReference, ModuleLayer>> modulesInLayer =
                    new ArrayList<>();
            for (ResolvedModule module : modulesInLayerSet) {
                modulesInLayer
                        .add(new SimpleEntry<>(module.reference(), layer));
            }
            // Sort modules in layer by name for consistency
            Collections.sort(modulesInLayer,
                    (e1, e2) -> e1.getKey().descriptor().name()
                            .compareTo(e2.getKey().descriptor().name()));
            // To be safe, dedup ModuleReferences, in case a module occurs in multiple
            // layers and reuses its ModuleReference (no idea if this can happen)
            for (Entry<ModuleReference, ModuleLayer> m : modulesInLayer) {
                if (addedModules.add(m.getKey())) {
                    moduleRefs.add(m);
                }
            }
        }
        return moduleRefs;
    }

    /** Get the classes in the call stack. */
    private static Class<?>[] getCallStack() {
        // Try StackWalker (JDK 9+)
        PrivilegedAction<Class<?>[]> stackWalkerAction =
                (PrivilegedAction<Class<?>[]>) () ->
                    StackWalker.getInstance(
                            Option.RETAIN_CLASS_REFERENCE)
                    .walk(s -> s.map(
                            StackFrame::getDeclaringClass)
                            .toArray(Class[]::new));
        try {
            // Try with doPrivileged()
            return AccessController
                    .doPrivileged(stackWalkerAction);
        } catch (Exception e) {
        }
        try {
            // Try without doPrivileged()
            return stackWalkerAction.run();
        } catch (Exception e) {
        }

        // Try SecurityManager
        PrivilegedAction<Class<?>[]> callerResolverAction = 
                (PrivilegedAction<Class<?>[]>) () ->
                    new SecurityManager() {
                        @Override
                        public Class<?>[] getClassContext() {
                            return super.getClassContext();
                        }
                    }.getClassContext();
        try {
            // Try with doPrivileged()
            return AccessController
                    .doPrivileged(callerResolverAction);
        } catch (Exception e) {
        }
        try {
            // Try without doPrivileged()
            return callerResolverAction.run();
        } catch (Exception e) {
        }

        // As a fallback, use getStackTrace() to try to get the call stack
        try {
            throw new Exception();
        } catch (final Exception e) {
            final List<Class<?>> classes = new ArrayList<>();
            for (final StackTraceElement elt : e.getStackTrace()) {
                try {
                    classes.add(Class.forName(elt.getClassName()));
                } catch (final Throwable e2) {
                    // Ignore
                }
            }
            if (classes.size() > 0) {
                return classes.toArray(new Class<?>[0]);
            } else {
                // Last-ditch effort -- include just this class
                return new Class<?>[] { Java9Scanner.class };
            }
        }
    }

    /**
     * Return true if the given module name is a system module.
     * There can be system modules in layers above the boot layer.
     */
    private static boolean isSystemModule(
            final ModuleReference moduleReference) {
        String name = moduleReference.descriptor().name();
        if (name == null) {
            return false;
        }
        return name.startsWith("java.") || name.startsWith("jdk.")
            || name.startsWith("javafx.") || name.startsWith("oracle.");
    }

    public static void main(String[] args) throws Exception {
        // Get ModuleReferences for modules of all classes in call stack,
        List<Entry<ModuleReference, ModuleLayer>> systemModuleRefs = new ArrayList<>();
        List<Entry<ModuleReference, ModuleLayer>> nonSystemModuleRefs = new ArrayList<>();

        Class<?>[] callStack = getCallStack();
        List<Entry<ModuleReference, ModuleLayer>> moduleRefs = findModuleRefs(
                callStack);
        // Split module refs into system and non-system modules based on module name
        for (Entry<ModuleReference, ModuleLayer> m : moduleRefs) {
            (isSystemModule(m.getKey()) ? systemModuleRefs
                    : nonSystemModuleRefs).add(m);
        }

        // List system modules
        System.out.println("\nSYSTEM MODULES:\n");
        for (Entry<ModuleReference, ModuleLayer> e : systemModuleRefs) {
            ModuleReference ref = e.getKey();
            System.out.println("  " + ref.descriptor().name());
        }

        // Show info for non-system modules
        System.out.println("\nNON-SYSTEM MODULES:");
        for (Entry<ModuleReference, ModuleLayer> e : nonSystemModuleRefs) {
            ModuleReference ref = e.getKey();
            ModuleLayer layer = e.getValue();
            System.out.println("\n  " + ref.descriptor().name());
            System.out.println(
                    "    Version: " + ref.descriptor().toNameAndVersion());
            System.out.println(
                    "    Packages: " + ref.descriptor().packages());
            System.out.println("    ClassLoader: "
                    + layer.findLoader(ref.descriptor().name()));
            Optional<URI> location = ref.location();
            if (location.isPresent()) {
                System.out.println("    Location: " + location.get());
            }
            try (ModuleReader moduleReader = ref.open()) {
                Stream<String> stream = moduleReader.list();
                stream.forEach(s -> System.out.println("      File: " + s));
            }
        }
    }
}
Luke Hutchison
  • 8,186
  • 2
  • 45
  • 40
  • 2
    Here are more detailed investigations by the same author: https://github.com/lukehutch/fast-classpath-scanner/issues/36 – Vadzim Oct 20 '17 at 17:36
  • 2
    @kaqqao I updated my answer with the complete scanning code. – Luke Hutchison Jun 13 '18 at 10:38
  • @LukeHutchison You sir, are the god-emperor of human kind :) Does this mean FastClasspathScanner is now able to deal with Java 9 with modulepath (and no classpath)? – kaqqao Jun 14 '18 at 13:51
  • `isSystemModule` seems problematic, esp with current EE classes (and the future ee4j `jakarta` namespace too) – Joakim Erdfelt Sep 24 '18 at 14:29
  • @JoakimErdfelt thanks for pointing this out. Can you give a few examples, please? Do you know any other way of detecting system modules? – Luke Hutchison Sep 25 '18 at 17:12
  • 1
    @LukeHutchison perhaps if the location url for the module has scheme `jrt:`? That means the module belongs to the java-runtime, right? – Joakim Erdfelt Sep 26 '18 at 12:08
  • @JoakimErdfelt actually I just discovered that for a jlink'd project, all URLs become `jrt:` URLs. I changed `isSystemModule()` back to use name-based criteria -- specifically the following are assumed to be system modules: `java.*`, `jdk.*`, `javafx.*`, and `oracle.*`. I left off `javafx.*`. Does that create a problem with any of the EE modules you were referring to? – Luke Hutchison Feb 10 '19 at 21:45
  • @LukeHutchison the entire Java EE namespace has moved to Eclipse. Which has rebranded as Jakarta EE, already a few modules on maven central with this new namespace - http://central.maven.org/maven2/jakarta/ – Joakim Erdfelt Feb 11 '19 at 15:19
  • @JoakimErdfelt thanks, that's helpful. But are you saying that all of `jakarta.*` should be treated as system modules? And are the module names the same as the package names? – Luke Hutchison Feb 12 '19 at 17:40
  • 2
    You can simplify the stack walker usage to a single `return AccessController.doPrivileged((PrivilegedAction[]>)() -> StackWalker.getInstance(Option.RETAIN_CLASS_REFERENCE) .walk(s -> s.map(StackFrame::getDeclaringClass).toArray(Class>[]::new)));` statement. No need to add to an `ArrayList` manually nor converting it to an array afterwards. And in Java 9 code, you can use lambda expressions (as you already do at some places), instead of anonymous inner classes. – Holger Aug 15 '19 at 17:55
  • @Holger thanks for the improvements! Except that your code doesn't compile unless you replace `Class>[]::new` with `Class[]::new` due to `Type mismatch: cannot convert from Class>[] to Class[]`. I'll update my code example though. – Luke Hutchison Aug 16 '19 at 20:01
  • When you still use the variable, you don’t need the type cast. In my example, I passed the lambda directly to `doPrivileged`, which requires the type cast for disambiguation. But with a local variable, its declared type does already the job. Don’t know why your compiler didn’t accept `.toArray(Class>[]::new)`. It should, and all versions of `javac`, I tried, did. – Holger Aug 19 '19 at 08:23
2

The actual issue here is to find the paths to all jars and folders on the classpath. Once when you have them, you can scan.

What I did is the following:

  • get the current module descriptor for current class
  • get all requires modules
  • for each such module open resource of MANIFEST.MF
  • remove the MANIFEST.MF path from the resource url
  • what remains is the classpath of the module, i.e. to it's jar or folder.

I do the same for current module, to get the classpath for current code.

This way I collect classpath of a currently working module and all its required modules (1 step away). That was working for me - and my Java8 scanner was still being able to do the job. This approach does not require any additional VM flag etc.

I could extend this approach to get all required modules easily (not only the first level), but for now, I don't need that.

Code.

igr
  • 10,199
  • 13
  • 65
  • 111