5

I have a conflict with a transitive dependency. Overriding, excluding or forcing does not help. What else can I do to get the right version of the library into the jar? The full code dan be. found https://github.com/geoHeil/gradle-dependency-resolution but the main parts of it are described below.

problem description

executing

./gradlew shadowJar

with geomesa dependencies (which pull in an outdated version of typesafe/lightbend configuration library) disabled:

dependencies {

    compile "com.github.kxbmap:configs_2.11:0.4.4"
    //compile "org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1"
}

and executing the jar

java -jar build/libs/gradleThing-all.jar

outputs

hello
my config is: Success(Job1Configuration(frequencyCounting))

When enabling the dependencies:

./gradlew shadowJar
java -jar build/libs/gradleThing-all.jar

it fails with

hello
Exception in thread "main" java.lang.Exception: Failed to start. There is a problem with the configuration: Vector([extract] com.typesafe.config.Config.hasPathOrNull(Ljava/lang/String;)Z)

which is related to an outdated version of the config library, how to resolve NoSuchMethodError on typesafe config?. Also confirmed via:

gradle dependencyInsight --dependency om.typesafe:config

> Task :dependencyInsight
com.typesafe:config:1.3.1 (conflict resolution)
   variant "runtime" [
      Requested attributes not found in the selected variant:
         org.gradle.usage = java-api
   ]
\--- com.github.kxbmap:configs_2.11:0.4.4
     \--- compileClasspath

com.typesafe:config:1.2.1 -> 1.3.1
   variant "runtime" [
      Requested attributes not found in the selected variant:
         org.gradle.usage = java-api
   ]
+--- org.locationtech.geomesa:geomesa-convert-avro_2.11:2.0.1
|    \--- org.locationtech.geomesa:geomesa-convert-all_2.11:2.0.1
|         +--- org.locationtech.geomesa:geomesa-tools_2.11:2.0.1
|         |    \--- org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1
|         |         \--- compileClasspath
...

How can I fix the transitive dependency to my desired version of 1.3.3?

my solutions

Trying to set:

compile("org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1") {
                exclude group: 'com.typesafe', module: 'config'
            }

and re running the jar fails again with the same issue.

Also an constraint https://docs.gradle.org/current/userguide/managing_transitive_dependencies.html#sec:dependency_constraints like:

implementation("com.typesafe:config")
    constraints {
        implementation("com.typesafe:config:1.3.3") {
            because 'previous versions miss a method https://stackoverflow.com/questions/40610816/how-to-resolve-nosuchmethoderror-on-typesafe-config'
        }
    }

or using force https://docs.gradle.org/current/userguide/managing_transitive_dependencies.html#sec:enforcing_dependency_version like : 

implementation('com.typesafe:config:1.3.3') {
        force = true
    }

or like:

configurations.all {
    resolutionStrategy {
        force 'com.typesafe:config:1.3.3'
    }
}

does not give the right version.

result

All variants fail with the same error

What is wrong? There must be a way to force the desired version.

Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
  • Grade 4.8 RC2 does not seem to help https://github.com/gradle/gradle/issues/5364 – Georg Heiler May 30 '18 at 11:46
  • What does `dependencyInsight` give after excluding through `exclude group: 'com.typesafe', module: 'config'` ? – ToYonos May 30 '18 at 12:08
  • https://gist.github.com/geoHeil/ca0d4670c1dcc13f1ed9c7429564c2a5 is returned, i.e. as expected only that dependency is visualized in the logs, however this is **NOT** the result I get in the jar, as otherwise the job would not have crashed. – Georg Heiler May 30 '18 at 12:46
  • Also the ideas of https://discuss.gradle.org/t/how-to-force-a-dependency-version-while-also-substituting-a-transitive-dependency/26759/7 do not work. – Georg Heiler May 30 '18 at 13:53

1 Answers1

2

The problem you have is the exact opposite of what you are describing.

The different things you tried in your project all make sure that the version of com.typesafe:config is a 1.3.x.

However, when adding org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1 you pull in some dependency that already has the classes from com.typesafe:config but from the 1.2 line. And when you create your fat jar, that version overwrites the classes from the 1.3 line.

This can be seen by decompressing your fat jar and running:

$ javap com/typesafe/config/Config.class | grep hasPath
  public abstract boolean hasPath(java.lang.String);

showing that indeed the hasPathOrNull method is missing.

That shading issue is hinted at in https://issues.apache.org/jira/browse/SPARK-9441.

Given this, the easy path - if possible for you - would be to downgrade "com.github.kxbmap:configs_2.11 to a version that relies on com.typesafe:config:1.2.x

The other solution is to find out exactly which fat jar is including these and see if you can exclude it from your own fat jar.

Louis Jacomet
  • 13,661
  • 2
  • 34
  • 43
  • Why does not even the exclude working? Should that not exclude the dependency from the fat jar? – Georg Heiler May 30 '18 at 17:30
  • My intention was to get rid of the 1.2 classes – Georg Heiler May 30 '18 at 17:43
  • What would the process look like to find the exact jar? Should `dependencyInsight ` be the tool of choice? – Georg Heiler May 31 '18 at 07:07
  • Any attempt to modify the dependencies does not work because the problematic version does not appear as a dependency. The classes are already _inside_ another dependency. – Louis Jacomet May 31 '18 at 07:46
  • Any chance to identify such a offending dependency with a tool? – Georg Heiler May 31 '18 at 07:48
  • You can try one of the methods described in http://www.baeldung.com/gradle-fat-jar and leverage the duplicate strategy from the Jar task, by making it fail on duplicates: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.bundling.Jar.html#org.gradle.api.tasks.bundling.Jar:duplicatesStrategy – Louis Jacomet May 31 '18 at 07:49