12

I'm trying to run MaxMind's geoIP in a spark task, but I'm getting a NoSuchMethodException from a maxmind call to a jackson-databind library. I've removed all other versions of jackson-databind using mvn exclusions, yet the error persists after a mvn clean. What am I missing?

No dependency conflicts:

mvn dependency:tree -Dverbose -Dincludes=com.fasterxml.jackson.core:jackson-databind ... [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ sift-etl --- [INFO] com.sift.etl:sift-etl:jar:0.1.6 [INFO] \- com.maxmind.geoip2:geoip2:jar:2.6.0:compile [INFO] \- com.maxmind.db:maxmind-db:jar:1.2.0:compile [INFO] \- com.fasterxml.jackson.core:jackson-databind:jar:2.7.0:compile

Error:

Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V at com.maxmind.db.Decoder.decodeArray(Decoder.java:272)

Edit: I have a ton of org.codehaus.jackson, do I need to exclude these as well? How can I resolve the differences in namespaces?

Edit: This is built into a scala based project that runs in Apache Spark. The jar is compiled into a shaded jar using maven-shade-plugin 2.4

Edit: Here's the complete output for jar tvf on the shaded jar: https://drive.google.com/file/d/0B2ZVKNsRXgTbeUdEU2ZhM2J3dmc/view?usp=sharing

Here's the result of a grep for jackson-databind: 0 Mon Jan 25 09:53:54 PST 2016 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/ 151 Mon Jan 25 09:53:54 PST 2016 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.properties 5192 Mon Jan 25 09:53:54 PST 2016 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml

and ArrayNode: /usr/lib/jvm/sift-jdk1.8.0_66/bin/jar -tvf target/sift-etl-0.1.6-shaded.jar | grep ArrayNode 15060 Sat Jan 23 01:33:14 UTC 2016 shaded/parquet/org/codehaus/jackson/node/ArrayNode.class 1309 Sat Jan 23 01:33:18 UTC 2016 org/apache/lucene/index/DocumentsWriterDeleteQueue$QueryArrayNode.class 1645 Sat Jan 23 01:33:18 UTC 2016 org/apache/lucene/index/DocumentsWriterDeleteQueue$TermArrayNode.class 18145 Sat Jan 23 01:33:20 UTC 2016 com/fasterxml/jackson/databind/node/ArrayNode.class 1058 Sat Jan 23 01:33:22 UTC 2016 org/apache/commons/configuration/plist/XMLPropertyListConfiguration$ArrayNode.class

Initializing and querying the maxmind GeoIP DB runs successfully in tests. The error only occurs in production when run as a spark task -- after it has been packaged using maven-shade.

maven-shade plugin config: http://pastebin.com/QzrhM5Ee

Is it meaningful that the shaded jar doesn't contain the jackson-databind jar, but all of the *.class files within the jar?

bhan
  • 2,489
  • 3
  • 19
  • 26
  • The problem seems to be related to the version of `jackson-databind` in the `spark-assembly.jar`. Here my _-verbose_ trace : `[Loaded org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1 from file:/home/vagrant/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar]` – Stan May 12 '16 at 12:36

3 Answers3

10

The version of the ArrayNode constructor that takes a JsonNodeFactory and a List<JsonNode> as parameters was removed as of version 2.2.0 of com.fasterxml.jackson.core:jackson-databind jar, so this looks like a bug in the maxmind-db library, since it is pulling in version 2.7.0 of jackson-databind but (at least indirectly) calling an obsolete method.

If possible, you could try downgrading your jackson-databind dependency by explicitly including in your pom file like so:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.1.0</version>
</dependency>
palimpsestor
  • 1,038
  • 1
  • 8
  • 28
  • 1
    It appears to me that this was [added in 2.7.0](https://github.com/FasterXML/jackson-databind/commit/12e811240e7f492a7a0639730c59c4a0cac10578#diff-ff1cff3f8cbf4dab0ac0fdd2435b1b87R36). – Greg Oschwald Jan 23 '16 at 03:53
  • I stand corrected. I guess that constructor was *re*added in 2.7.0, as it was [extant in 2.1.0 and then disappeared in 2.2.0](http://grepcode.com/file_/repo1.maven.org/maven2/com.fasterxml.jackson.core/jackson-databind/2.2.0/com/fasterxml/jackson/databind/node/ArrayNode.java/?v=diff&id2=2.1.0). – palimpsestor Jan 23 '16 at 04:41
  • Given that, it seems that despite the maven dependency, that version of `jackson-databind` is not in the classpath of whatever classloader is executing the code in question. – palimpsestor Jan 23 '16 at 04:51
  • Hmm, is there any thing else I can provide/debug to find the cause of the issue? I'm kind of at a loss on what else to look at. For what its worth, this is a scala based project and I am trying to use maxmind within an Apache Spark task. – bhan Jan 24 '16 at 04:43
  • If you are running a Spark task, does that mean you are preparing a "fat jar" containing all the dependencies? If so, is version 2.7.0 of the `jackson-databind` jar contained in the fat jar? – palimpsestor Jan 24 '16 at 06:04
  • Is a shaded jar the same thing? If so, yes, we are dumping everything into a single "shaded" jar. How does this affect the dependency tree? Mvn still shows no dependency conflicts for com.fasterxml.jackson.core:jackson-databind 2.7.0 – bhan Jan 25 '16 at 17:12
  • Added more detail regarding the jar in the question. – bhan Jan 25 '16 at 17:43
  • Yes, "fat jar", "shaded jar", and "uber jar" are all informally synonymous. Can you share the result of `jar tvf` on that shaded jar? – palimpsestor Jan 25 '16 at 17:45
  • Here's the complete result (10MB): https://drive.google.com/file/d/0B2ZVKNsRXgTbeUdEU2ZhM2J3dmc/view?usp=sharing – bhan Jan 25 '16 at 18:04
  • Here's the result of a grep for jackson-databind: ``` 0 Mon Jan 25 09:53:54 PST 2016 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/ 151 Mon Jan 25 09:53:54 PST 2016 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.properties 5192 Mon Jan 25 09:53:54 PST 2016 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml ``` – bhan Jan 25 '16 at 18:04
  • ```jar -tvf sift-etl-0.1.6-shaded.jar | grep ArrayNode 15060 Sat Jan 23 01:33:14 UTC 2016 shaded/parquet/org/codehaus/jackson/node/ArrayNode.class 1309 Sat Jan 23 01:33:18 UTC 2016 org/apache/lucene/index/DocumentsWriterDeleteQueue$QueryArrayNode.class 1645 Sat Jan 23 01:33:18 UTC 2016 org/apache/lucene/index/DocumentsWriterDeleteQueue$TermArrayNode.class 18145 Sat Jan 23 01:33:20 UTC 2016 com/fasterxml/jackson/databind/node/ArrayNode.class 1058 Sat Jan 23 01:33:22 UTC 2016 org/apache/commons/configuration/plist/XMLPropertyListConfiguration$ArrayNode.class ``` – bhan Jan 25 '16 at 18:11
  • Can you also `jar xvf sift-etl-0.1.6-shaded.jar META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml` and inspect that pom to verify that it is indeed version 2.7.0 of `jackson-databind. Also might help to share the section of your pom where you are configuring the maven shade plugin. – palimpsestor Jan 25 '16 at 18:45
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/101617/discussion-between-bhan-and-palimpsestor). – bhan Jan 25 '16 at 19:17
5

Honestly, I couldn't get this working with the <relocation> to the shaded jar config or from downgrading jackson-databind, so I ended up just downgrading com.maxmind.geoip2 as was the latest suggestion from the Github issue thread that led me here, and it worked!

<!-- Using an old version b/c https://github.com/maxmind/GeoIP2-java/issues/57 -->
<dependency>
  <groupId>com.maxmind.geoip2</groupId>
  <artifactId>geoip2</artifactId>
  <version>2.5.0</version>
</dependency>
Mr. Kevin Thomas
  • 837
  • 2
  • 13
  • 21
2

This was resolved by adding the following relocation to the shaded jar config:

<relocation> <pattern>com.fasterxml.jackson.core</pattern> <shadedPattern>com.shaded.fasterxml.jackson.core</shadedPattern> </relocation>

bhan
  • 2,489
  • 3
  • 19
  • 26
  • I'm facing the same issue while trying to use Maxmind GeoIP2 with Spark - Scala. Could you elaborate on your solution a bit? Did you recompile Spark with the above shade config or is it something else? – Sai Kiriti Badam Nov 22 '17 at 12:13