1

When I ran with spark-submit for the following simple Spark program of:

import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext
import org.apache.spark._
import SparkContext._

object TEST2{
def main(args:Array[String])
{
  val conf = new SparkConf().setAppName("TEST")
  val sc=new SparkContext(conf)

  val list=List(("aa",1),("bb",2),("cc",3))
  val maps=list.toMap
}

}

I got java.lang.NoSuchMethodError for the line of "val maps=list.toMap". But in a spark-shell or simply scala, it has no problem:

scala> val list=List(("aa",1),("bb",2),("cc",3))
list: List[(String, Int)] = List((aa,1), (bb,2), (cc,3))

scala> val maps=list.toMap
maps: scala.collection.immutable.Map[String,Int] = Map(aa -> 1, bb -> 2, cc -> 3)

So to use "toMap" method, what am I missing in spark-submit? I use "sbt package" to compile the program and without problem. Thanks!

P.S: the build.sbt file is as:

name := "TEST2"
version := "1.0"
scalaVersion := "2.11.6"
user1460691
  • 61
  • 1
  • 7
  • 2
    Perhaps you could include the full error message in the question. – Daniel Darabos Jul 23 '15 at 21:50
  • 1
    i executed your code against my spark-1.2.0 &scala 2.10.4 , it worked without any errors on spark-submit. try scalaVersion := "2.10.4" in sbt. Note : till date spark-all-versions are buit against 2.10.4, not 2.11.x – vijay kumar Jul 24 '15 at 08:07
  • Changed to 2.10.4 but still the same. Did you set specially to your CLASSPATH? But "toMap" is a so basic method in scala, quite weird that it could be found by spark-submit. I tested on each node that this method works fine. – user1460691 Jul 24 '15 at 14:26
  • I had the same problem on Spark 1.3.0 (CDH 5.4.4) but setting the scala version to 2.10.4 _did_ fix the problem for me. – Brent Miller Aug 10 '15 at 04:31

1 Answers1

0

sbt package creates a JAR containing only your project, not all its dependencies.

When using spark-submit, you need a JAR that contains your project and all dependencies except Spark libraries (already provided by the Spark installation on your cluster)

This uber-JAR can be created using the sbt assembly plugin:

  • In your project folder, create the file project/assembly.sbt with this line :

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")

  • In your build.sbt, the Spark dependencies should be tagged as "provided":

    libraryDependencies ++= Seq(
  ...
  "org.apache.spark"  %% "spark-core" % sparkVersion % "provided"
  ...
)

  • Run sbt assembly, which will create the uber-JAR target/scala-2.11/TEST2-1.0.jar

Didier
  • 430
  • 6
  • 15