Executing Spark scala program after compilation

Question

I have compiled Spark scala program on command line. But now I want to execute it. I dont want to use Maven or sbt. the program .I have used the command to execute the

scala -cp ".:sparkDIrector/jars/*" wordcount

But I am getting this error java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

import org.apache.spark._
import org.apache.spark.SparkConf

/** Create a RDD of lines from a text file, and keep count of
 *  how often each word appears.
 */
object wordcount1 {

  def main(args: Array[String]) {
      // Set up a SparkContext named WordCount that runs locally using
      // all available cores.

      println("before conf")
      val conf = new SparkConf().setAppName("WordCount")
      conf.setMaster("local[*]")
      val sc = new SparkContext(conf)
      println("after the textfile")

      // Create a RDD of lines of text in our book
      val input = sc.textFile("book.txt")

      println("after the textfile")
      // Use flatMap to convert this into an rdd of each word in each line
      val words = input.flatMap(line => line.split(' '))
      // Convert these words to lowercase
      val lowerCaseWords = words.map(word => word.toLowerCase())
      // Count up the occurence of each unique word

      println("before text file")
      val wordCounts = lowerCaseWords.countByValue()

      // Print the first 20 results
      val sample = wordCounts.take(20)

      for ((word, count) <- sample) {
        println(word + " " + count)
      }

      sc.stop()
    }
}

It is showing that the error is at location

val conf = new SparkConf().setAppName("WordCount").

Any help?

Answer 1

Starting from Spark 2.0 the entry point is the SparkSession:

import org.apache.spark.sql.SparkSession
val spark = SparkSession
  .builder
  .appName("App Name")
  .getOrCreate()

Then you can access the SparkContext and read the file with:

spark.sparkContext().textFile(yourFileOrURL)

Remember to stop your session at the end:

spark.stop()

I suggest you to have a look at these examples: https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples

Then, to launch your application, you have to use spark-submit:

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

In your case, it will be something like:

./bin/spark-submit \
  --class wordcount1 \
  --master local \
  /path/to/your.jar