I Build Scala Project in Intellij and I try to run an Apache Spark example. Here is my code :
package sparkTraining
import org.apache.spark.{SparkConf, SparkContext}
object SimpleApp {
def main(args: Array[String]) {
val logFile = "/usr/local/spark/README.md"
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println(s"Lines with a: $numAs, Lines with b: $numBs")
sc.stop()
}
}
I have installed Scala and SBT plugin on Intellij and here is my build.sbt code:
name := "sparkTraining"
version := "1.0"
scalaVersion := "2.12.0"
and I added all Saprk 2.0.1 jars to the project. But the problem is when I run this piece of code it errors:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
at org.apache.spark.util.Utils$.getCallSite(Utils.scala:1410)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:77)
at sparkTraining.SimpleApp$.main(testScala.scala:9)
at sparkTraining.SimpleApp.main(testScala.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
I searched for it and I couldn't find any useful answer.
Here is some points:
- I have two user, user1 and hduser, and all my Big Data project like Hadoop, Spark and Scala are under hduser and all paths are set in hduser .bashrc, while Intellij is under root (in /opt folder).
- while executing this code Spark is not running in my machine.
These two points have any affect on executing? How can I solved my problem ? Thanks
