1

I have used Spark before in yarn-cluster mode and it's been good so far.

However, I wanted to run it "local" mode, so I created a simple scala app, added spark as dependency via maven and then tried to run the app like a normal application.

However, I get the above exception in the very first line where I try to create a SparkConf object.

I don't understand, why I need hadoop to run a standalone spark app. Could someone point out what's going on here.

My two line app:

 val sparkConf = new SparkConf().setMaster("local").setAppName("MLPipeline.AutomatedBinner")//.set("spark.default.parallelism", "300").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.mb", "256").set("spark.akka.frameSize", "256").set("spark.akka.timeout", "1000") //.set("spark.akka.threads", "300")//.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") //.set("spark.akka.timeout", "1000")
 val sc = new SparkContext(sparkConf)
zero323
  • 322,348
  • 103
  • 959
  • 935
MV23
  • 285
  • 5
  • 17
  • see this link for an answer http://stackoverflow.com/questions/30906412/noclassdeffounderror-com-apache-hadoop-fs-fsdatainputstream-when-execute-spark-s – Abdulrahman Aug 28 '15 at 02:14
  • http://stackoverflow.com/questions/30906412/noclassdeffounderror-com-apache-hadoop-fs-fsdatainputstream-when-execute-spark-s this mat help you! – YOung Jan 13 '16 at 07:59

0 Answers0