spark-submit not finding my class declarations with java.lang.ClassNotFoundException

Question

I have been running Spark with Scala locally without any problem on my Mac, and then one day, it suddenly cant find my class declaration. It compiles just fine with maven, and when I run spark-submit, no matter what I change, it ALWAYS responds as below. I have stripped out all packages and just running the most bare-bones example app. Any ideas what to test? It appears the path is somehow broken. Scala code runner version 2.10.4, java version "1.7.0_79", Spark version 1.5.1

$ /usr/local/spark-1.5.1/bin/spark-submit --class “SimpleApp” --master local target/simplesparkproject-0.0.1.jar 
java.lang.ClassNotFoundException: “SimpleApp”
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:639)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Here is the snippet from the scala script:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {  
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    println("hello world")
  }
}

And here is the file structure:

$ find .
.
./.DS_Store
./pom.xml
./README.txt
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala
./target
./target/classes
./target/classes/SimpleApp$.class
./target/classes/SimpleApp.class
./target/classes.335735219.timestamp
./target/maven-archiver
./target/maven-archiver/pom.properties
./target/simplesparkproject-0.0.1.jar

score 0 · Answer 1 · edited May 23 '17 at 10:27

0

Did you tried with sbt ?

I got the same error and with this answer i solved :

java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags

edited May 23 '17 at 10:27

Community

1
1

answered Nov 11 '15 at 16:11

I did try both sbt and maven and get identical errors – claudiaann1 Nov 11 '15 at 16:14

score 0 · Answer 2 · answered Nov 11 '15 at 21:31

0

find . doesn't show what's inside your jar.

Run jar tf simplesparkproject-0.0.1.jar for viewing the contents of a JAR file

answered Nov 11 '15 at 21:31

blockR

1,937
1
11
8

$ jar tf target/simplesparkproject-0.0.1.jar META-INF/ META-INF/MANIFEST.MF SimpleApp$.class SimpleApp.class META-INF/maven/ META-INF/maven/myGroupID/ META-INF/maven/myGroupID/simplesparkproject/ META-INF/maven/myGroupID/simplesparkproject/pom.xml META-INF/maven/myGroupID/simplesparkproject/pom.properties – claudiaann1 Nov 11 '15 at 21:48
Its showing up in the jar file, apparently. And if I put the full path to the jar, same thing.... – claudiaann1 Nov 12 '15 at 00:46

score 0 · Accepted Answer · answered Nov 12 '15 at 18:17

0

The only way I have been able to solve the problem is by re-installing spark and then modifying my .bashrc to no longer point directly to Scala. After doing these things, it worked as expected.

answered Nov 12 '15 at 18:17

claudiaann1

237
3
12

spark-submit not finding my class declarations with java.lang.ClassNotFoundException

3 Answers3