3

I try to use https://github.com/snowplow/scala-maxmind-iplookups library in AWS Glue job. I use https://github.com/sbt/sbt-assembly to create jar with all dependencies.

In the case when jackson is included

dependencyOverrides ++= Seq(
  "com.fasterxml.jackson.core" % "jackson-core" % "2.9.3",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.3",
  "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.3",
)

the error is

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.3

full log https://gist.github.com/pawaclawczyk/81844b5063d998acd3528f136c7a01f5

In case when jackson is excluded

excludeDependencies ++= Seq(
  ExclusionRule("com.fasterxml.jackson.core")
)

the error is

ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V

full log https://gist.github.com/pawaclawczyk/825d66e2148d688e274eb36c99c18a89

pwc
  • 446
  • 3
  • 10

1 Answers1

0

AWS Glue brings it's on Jackson dependencies, I am not sure which version though. You can check the glue-assembly.jar contents - here's how to get it.

The problem is that your assembly and glue-assembly both end up on same classpath on Spark executor.

I have had the same problem with incompatible libraries in my project when using Google Big Query Connector (mainly jackson and guava) and I solved it by shading these libraries in my assembly jar like this:

lazy val root = (project in file(".")).
  settings(
    ...,
    assemblyShadeRules in assembly := Seq(
      ShadeRule.rename("com.fasterxml.**" -> "my.shaded.@0").inAll,
      // other shade rules
    ),
    libraryDependencies ++= ...
  )
botchniaque
  • 4,698
  • 3
  • 35
  • 63