Caused by: java.lang.NoClassDefFoundError: scala/collection/GenMap

Question

java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider com.fasterxml.jackson.module.scala.DefaultScalaModule could not be instantiated
    at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:804)
    at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:722)
    at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395)
    at com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1131)
    at org.apache.beam.sdk.testing.TestPipeline.<clinit>(TestPipeline.java:253)

. . .

Caused by: java.lang.NoClassDefFoundError: scala/collection/GenMap
    at com.fasterxml.jackson.module.scala.modifiers.ScalaTypeModifier.<init>(ScalaTypeModifier.scala:15)
    at com.fasterxml.jackson.module.scala.modifiers.ScalaTypeModifierModule.$init$(ScalaTypeModifier.scala:54)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:18)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780)
    ... 61 more

Nowhere in our code do we directly call GenMap but it seems to be a side effect of calling jackson in our tests.

The error happens at

public final transient TestPipeline pipeline = TestPipeline.create();

specifically org.apache.beam.sdk.testing.TestPipeline in

@RunWith(JUnit4.class)
public class BeamUtilsTest {

    private static final Logger logger = LoggerFactory.getLogger(BeamUtilsTest.class);

    @Rule
    public final transient TestPipeline pipeline = TestPipeline.create();

    @Test
    public void testJoins() throws Exception {
        Map<String, PCollection<JsonNode>> map = new HashMap<>();
        map.put("users", pipeline.apply("readUsers",
                                    BeamUtils.readJSONFile(pathOfResource("join-data/users.jsonl"))));
        map.put("accounts", pipeline.apply("readAccounts",
                                       BeamUtils.readJSONFile(
                                               pathOfResource("join-data/accounts.jsonl"))));

        JsonNode joinExpression = JsonFile.read(pathOfResource("join-data/usersAccounts.json"));
        PCollection<JsonNode> joinedCollection = BeamUtils.join(map, JsonBuilder.newBuilder().set("/withJoin",joinExpression).build());
        Map<String, String> validationMap = new HashMap<>();
        validationMap.put("mickey.mouse", "AD01");
        validationMap.put("minni.mouse", "AD05");
        validationMap.put("donald.duck", "AD06");
        PAssert.that("Join fields should exist", joinedCollection)
           .satisfies((SerializableFunction<Iterable<JsonNode>, Void>) jsonNodeIterable -> {
               jsonNodeIterable.forEach(jsonNode -> {
                   logger.info(jsonNode.toPrettyString());
                   if (!jsonNode.path("userId").isMissingNode()) {
                       String userId = jsonNode.path("userId").textValue();
                       assert !validationMap.containsKey(userId) || jsonNode.path("accountId").textValue() != null;
                   }
               });
               return (Void) null;
           });
        pipeline.run();
    }

Context

I am trying to upgrade our project from Java 8 and older to Java 11 and 17, where most of my time has been spent in dependency hell, especially dealing with the various base version of Scala, old Apache Spark code, and other things.

This is simply the latest problem I am trying to resolve, but I have been fighting it for so long, I am not clear on backing out of the problem.

The test context is various services running under Docker. While I have some experience with Beam, and using dealing with JSON in Beam, I did not write this test, and I am a little mystified why jackson needs Scala libraries.

At this point, I would appreciate some insights as I am getting a little brain-dead working on this issue, and losing my ability to think creatively.

I noticed that scala.collection.GenMap was removed in newer versions of Scala, so this may also be a contributing factor. Adding

testImplementation 'org.scala-lang:scala-library:2.12.9'    // scala.collection.GenMap
. . .
testFixturesImplementation 'org.scala-lang:scala-library:2.12.9'    // scala.collection.GenMap

to our build.gradle file does not seem to help.

spark has [jackson-module-scala](https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala) as dependency. I guess the error is related with that. Not all [scala releases are comptaible with jdk 17](https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html). Spark supports jdk 17 since 3.3.0. Without knowing which version of each library are you using is hard to know what is causing the issue. Is it possible to provide a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example)? — Gastón Schabas, Jul 06 '23 at 23:48
If you generate the complete dependency tree, there's a good chance you find the conflicts. Otherwise share it with us (mainly where Scala, Spark and Jackson appear). — Gaël J, Jul 07 '23 at 06:14
@GastónSchabas At this point, I am settling on Java 11 for a target, especially with respect to Scala, Spark, Beam, etc. A minimal reproducible example can be very difficult to engineer, but I will think about it. — Eric Kolotyluk, Jul 07 '23 at 16:52
@GaëlJ I have been trying to fathom the dependency tree in IntelliJ, but I cannot seem to get any insights. Perhaps I need to use other tooling to explore the dependency tree... — Eric Kolotyluk, Jul 07 '23 at 16:54
you can use [sbt dependencytree plugin](https://www.baeldung.com/scala/sbt-dependency-tree) to provide the dependency tree. Based on which sbt version you are using is how the plugin should be installed — Gastón Schabas, Jul 07 '23 at 17:05
My bad. You are using `gradle`. Not sure how you can print that like in sbt — Gastón Schabas, Jul 07 '23 at 17:06
just with a quick search, I found [View and Debug Dependencies](https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html) and the question [Using Gradle to find dependency tree](https://stackoverflow.com/questions/21645071/using-gradle-to-find-dependency-tree). I guess that could help — Gastón Schabas, Jul 07 '23 at 17:21
Okay, I figured out how to search for dependencies in IntelliJ... sometimes things are not obvious in the GUI. Also, thanks to @GastónSchabas I used Gradle. Neither tool can find GenMap, and that does not surprise me because the error does not show up until Beam is getting invoked with TestPipeline.create(); Based on my Beam experience, troubleshooting can be very frustrating, and this seems to be yet another example. — Eric Kolotyluk, Jul 07 '23 at 21:54

Caused by: java.lang.NoClassDefFoundError: scala/collection/GenMap

Context

0 Answers0