1

I want to check my scala spark code by REPL.
This code works well in spark-shell, but local scala with sbt didn't work.
(After executing sbt in the command line, and execute console command to start REPL.)

What should I do?

code

import org.apache.spark.SparkContext  
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

val conf = new SparkConf().setAppName("RddSample").setMaster("local[*]")
val sc = new SparkContext(conf)
val data = (0 to 10).toArray
val inputRDD = sc.parallelize(data)

error 

com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.9.4 requires Jackson Databind version >= 2.9.0 and < 2.10.0 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
  at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:18)
  at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:692)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:821)
... 40 elided

build.sbt

name := "Sratup"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "2.3.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % sparkVersion
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
libraryDependencies +="com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.4"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.4"
SCouto
  • 7,808
  • 5
  • 32
  • 49
k_trader
  • 43
  • 2
  • 10

1 Answers1

0

That Spark version, (in fact, AFAIK, every version up from 2.1.0 )already includes com.fasterxml.jackson.core so you don't need to put them in your build.sbt.

If you want to use a different version of them you should remove them from libraryDependencies and include them in dependencyOverrides

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala" % "2.9.4"

You can check which dependencies the spark 2.3.0 includes here

SCouto
  • 7,808
  • 5
  • 32
  • 49
  • 1
    I extracted the jackson*.jar from the jar folder of spark and copied it to the extended jar folder of sbt. This code works very well. Thank you so much! – k_trader Mar 27 '18 at 14:57