1

I have a sbt projects and I want to make a test with scala test and shared spark session. Several weeks ago my project started to make an error.

java.lang.ExceptionInInitializerError
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
.....
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)

There is a very simple test

import org.apache.spark.sql.QueryTest.checkAnswer
import org.apache.spark.sql.Row
import org.apache.spark.sql.test.SharedSparkSession

class SparkTestSpec extends SharedSparkSession  {
  import testImplicits._
  test("join - join using") {
    val df = Seq(1, 2, 3).toDF("int")

    checkAnswer(df, Row(1) :: Row(2) :: Row(3) :: Nil)
  }
}

And sbt config

ThisBuild / scalaVersion := "2.12.10"
val sparkVersion = "3.1.0"
val scalaTestVersion = "3.2.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion % Test,
  "org.apache.spark" %% "spark-sql" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-catalyst" % sparkVersion % Test,
  "org.apache.spark" %% "spark-catalyst" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-hive" % sparkVersion % Test,
  "org.apache.spark" %% "spark-hive" % sparkVersion % Test classifier "tests",
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-core" % sparkVersion % Test classifier "tests",
  "log4j" % "log4j" % "1.2.17",
  "org.slf4j" % "slf4j-log4j12" % "1.7.30",

  "org.scalatest" %% "scalatest" % scalaTestVersion % Test,
  "org.scalatestplus" %% "scalacheck-1-14" % "3.2.2.0",

)
Gaël J
  • 11,274
  • 4
  • 17
  • 32
Vadim
  • 753
  • 8
  • 22
  • 1
    I think the real question is "why are you using scala 2.10?" Its last maintenance release was over 6 years ago. You should ideally be on 2.13 or 3.x. If you switch, the problem should go away. – Dylan Feb 15 '23 at 16:58
  • @Dylan 3.0 is not officially supported by Spark although there are https://github.com/VirtusLab/iskra https://github.com/vincenzobaz/spark-scala3 https://github.com/zio/zio-quill/tree/master/quill-spark/src – Dmytro Mitin Feb 15 '23 at 17:34
  • 2
    https://stackoverflow.com/questions/71243144/scala-module-2-8-11-requires-jackson-databind-version-2-8-0-and-2-9-0 https://stackoverflow.com/questions/67769492/scala-module-2-12-3-requires-jackson-databind-version-2-12-0-and-2-13-0-but https://stackoverflow.com/questions/64563502/scala-module-requiring-specific-version-of-data-bind-for-spark https://stackoverflow.com/questions/64874383/elastic4s-scala-module-requires-jackson-databind-version https://stackoverflow.com/questions/73476352/getting-incompatible-jackson-version-while-reading-a-file-in-spark-scala – Dmytro Mitin Feb 15 '23 at 17:39
  • @Dylan According to OP's `build.sbt`, 2.12 is used, not 2.10. Probably 2.10 is from some dependencies – Dmytro Mitin Feb 15 '23 at 17:48
  • Can't reproduce. This test compiles and runs for me with this `build.sbt` – Dmytro Mitin Feb 15 '23 at 17:57
  • How do you run the test? `sbt clean compile test` – Dmytro Mitin Feb 15 '23 at 17:59
  • 1
    @DmytroMitin Thank you. "org.apache.spark" %% "spark-sql" % sparkVersion exclude("com.fasterxml.jackson.core", "jackson-databind") helped – Vadim Feb 17 '23 at 16:06

3 Answers3

4

This is a very classic issue with Jackson. The error tells you that you need to have a single version of Jackson across all your dependencies but it's not the case.

Usually you have both Spark and another library pulling transitively Jackson in different versions.

What you need to do is:

  • run sbt dependencyTree to identify which library is pulling Jackson and which version
  • define a dependencyOverrides to force the same Jackson version for all Jackson libraries (which version is up to you depending on compatibility with the other libraries needing it)
Gaël J
  • 11,274
  • 4
  • 17
  • 32
0

I got the same issue. Added below in build.sbt file, then it started compiling for me.

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.10.0"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.10.0"

https://github.com/FasterXML/jackson-module-scala/issues/513

Bhargav Velisetti
  • 5
  • 2
0

Building on @bhargav-velisetti's answer, I added all of the following

  dependencyOverrides ++= Seq(
    "com.fasterxml.jackson.module" % "jackson-module-scala_2.13" % "2.14.2",
    "com.fasterxml.jackson.datatype" % "jackson-datatype-jdk8" % "2.14.2",
    "com.fasterxml.jackson.datatype" % "jackson-datatype-jsr310" % "2.14.2",
    "com.fasterxml.jackson.dataformat" % "jackson-dataformat-cbor" % "2.14.2",
    "com.fasterxml.jackson.module" % "jackson-module-parameter-names" % "2.14.2",
    "com.fasterxml.jackson.core" % "jackson-databind" % "2.14.2",
    "com.fasterxml.jackson.core" % "jackson-core"     % "2.14.2"
  )

before discovering that just the first one is necessary (wasn't the first one I added!):

  dependencyOverrides ++= Seq(
    "com.fasterxml.jackson.module" % "jackson-module-scala_2.13" % "2.14.2",
  )
bwbecker
  • 1,031
  • 9
  • 21