Add custom folders to classpath in bazel java tests

Question

I'm trying to migrate a large codebase from maven to bazel and I've found that some of the tests write to target/classes and target/test-classes and the production code reads it as resources on the classpath. This is because maven surefire/failsafe run by default from the module directory and add target/classes and target/test-classes to the classpath. For me to migrate this large codebase the only reasonable solution is to create target, target/classes and target/test-classes folders and add the last two to the classpath of the tests.
Any ideas on how this can be achieved?

Thanks

Answer 1

Another line of approach. Instead of generating a test suite, create a custom javaagent and a custom class loader. Use jvm_flags to setup and configure it.

The javaagent has a premain method. This sounds like a natural place to do things that happen before the regular main method, even if they don't have anything to do with class instrumentation, debugging, coverage gathering, or any other usual uses of javaagents.

The custom javaagent reads system property extra.dirs and creates directories specified there. It then reads property extra.link.path and creates the symbolic links as specified there, so I can place resources where the tests expect them, without having to copy them.

Classloader is needed so that we can amend the classpath at runtime without hacks. Great advantage is that this solution works on Java 10.

The custom classloader reads system property extra.class.path and (in effect) prepends it before what is in java.class.path.

Doing things this way means that standard bazel rules can be used.

BUILD

runtime_classgen_dirs = ":".join([
            "target/classes",
            "target/test-classes",
])
java_test(
    ...,
    jvm_flags = [
        # agent
        "-javaagent:$(location //tools:test-agent_deploy.jar)",
        "-Dextra.dirs=" + runtime_classgen_dirs,
        # classloader
        "-Djava.system.class.loader=ResourceJavaAgent",
        "-Dextra.class.path=" + runtime_classgen_dirs,
    ],
    ,,,,
    deps = [
        # not runtime_deps, cause https://github.com/bazelbuild/bazel/issues/1566
        "//tools:test-agent_deploy.jartest-agent_deploy.jar"
    ],
    ...,
)

tools/BUILD

java_binary(
    name = "test-agent",
    testonly = True,
    srcs = ["ResourceJavaAgent.java"],
    deploy_manifest_lines = ["Premain-Class: ResourceJavaAgent"],
    main_class = "ResourceJavaAgent",
    visibility = ["//visibility:public"],
)

tools/ResourceJavaAgent.java

import java.io.File;
import java.io.IOException;
import java.lang.instrument.Instrumentation;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;

// https://stackoverflow.com/questions/60764/how-should-i-load-jars-dynamically-at-runtime
public class ResourceJavaAgent extends URLClassLoader {
    private final ClassLoader parent;

    public ResourceJavaAgent(ClassLoader parent) throws MalformedURLException {
        super(buildClassPath(), null);
        this.parent = parent; // I need the parent as backup for SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
        System.out.println("initializing url classloader");
    }

    private static URL[] buildClassPath() throws MalformedURLException {
        final String JAVA_CLASS_PATH = "java.class.path";
        final String EXTRA_CLASS_PATH = "extra.class.path";
        List<String> paths = new LinkedList<>();
        paths.addAll(Arrays.asList(System.getProperty(EXTRA_CLASS_PATH, "").split(File.pathSeparator)));
        paths.addAll(Arrays.asList(System.getProperty(JAVA_CLASS_PATH, "").split(File.pathSeparator)));
        URL[] urls = new URL[paths.size()];
        for (int i = 0; i < paths.size(); i++) {
            urls[i] = Paths.get(paths.get(i)).toUri().toURL(); // important only for resource url, really: this url must be absolute, to pass getClass().getResource("/users.properties").toURI()) with uri that isOpaque == false.
//            System.out.println(urls[i]);
        }
        // this is for spawnVM functionality in tests
        System.setProperty(JAVA_CLASS_PATH, System.getProperty(EXTRA_CLASS_PATH, "") + File.pathSeparator + System.getProperty(JAVA_CLASS_PATH));
        return urls;
    }

    @Override
    public Class<?> loadClass(String s) throws ClassNotFoundException {
        try {
            return super.loadClass(s);
        } catch (ClassNotFoundException e) {
            return parent.loadClass(s);  // we search parent second, not first, as the default URLClassLoader would
        }
    }

    private static void createRequestedDirs() {
        for (String path : System.getProperty("extra.dirs", "").split(File.pathSeparator)) {
            new File(path).mkdirs();
        }
    }

    private static void createRequestedLinks() {
        String linkPaths = System.getProperty("extra.link.path", null);
        if (linkPaths == null) {
            return;
        }
        for (String linkPath : linkPaths.split(",")) {
            String[] fromTo = linkPath.split(":");
            Path from = Paths.get(fromTo[0]);
            Path to = Paths.get(fromTo[1]);
            try {
                Files.createSymbolicLink(from.toAbsolutePath(), to.toAbsolutePath());
            } catch (IOException e) {
                throw new IllegalArgumentException("Unable to create link " + linkPath, e);
            }
        }
    }

    public static void premain(String args, Instrumentation instrumentation) throws Exception {
        createRequestedDirs();
        createRequestedLinks();
    }
}

Answer 2

If you could tell the tests where to write these files (in case target/classes and target/test-classes are hardcoded), and then turn the test run into a genrule, then you can specify the genrule's outputs as data for the production binary's *_binary rule.

Answer 3

I solved the first part, creating the directories. I still don't know how to add the latter two to classpath.

Starting from https://gerrit.googlesource.com/bazlets/+/master/tools/junit.bzl, I modified it to read

_OUTPUT = """import org.junit.runners.Suite;
import org.junit.runner.RunWith;
import org.junit.BeforeClass;
import java.io.File;
@RunWith(Suite.class)
@Suite.SuiteClasses({%s})
public class %s {
    @BeforeClass
    public static void setUp() throws Exception {
      new File("./target").mkdir();
    }
}
"""
_PREFIXES = ("org", "com", "edu")
# ...

I added the @BeforeClass setUp method.

I stored this as junit.bzl into third_party directory in my project.

Then in a BUILD file,

load("//third_party:junit.bzl", "junit_tests")

junit_tests(
    name = "my_bundled_test",
    srcs = glob(["src/test/java/**/*.java"]),
    data = glob(["src/test/resources/**"]),
resources = glob(["src/test/resources/**"]),
tags = [
    # ...
],
    runtime_deps = [
        # ...
    ],
],
    deps = [
        # ...
    ],
)

Now the test itself is wrapped with a setUp method which will create a directory for me. I am not deleting them afterwards, which is probably a sound idea to do.

The reason I need test resources in a directory (as opposed to in a jar file, which bazel gives by default) is that my test passes the URI to new FileInputStream(new File(uri)). If the file resides in a JAR, the URI will be file:/path/to/my.jar!/my.file and the rest of the test cannot work with such URI.