Apache Spark error

Question

Here's the error I'm getting:

 "C:\Program Files (x86)\Java\jdk1.8.0_91\bin\java" -Didea.launcher.port=7535 "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.1.2\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\charsets.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\deploy.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\access-bridge-32.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\cldrdata.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\dnsns.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\jaccess.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\jfxrt.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\localedata.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\nashorn.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\sunec.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\sunjce_provider.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\sunmscapi.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\sunpkcs11.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\ext\zipfs.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\javaws.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\jce.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\jfr.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\jfxswt.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\jsse.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\management-agent.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\plugin.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\resources.jar;C:\Program Files (x86)\Java\jdk1.8.0_91\jre\lib\rt.jar;F:\IJ\scatest\aas-master\ch02-intro\target\classes;C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.1.2\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain com.cloudera.datascience.intro.RunIntro
 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/rdd/RDD
    at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:125)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.rdd.RDD
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more

Process finished with exit code 1

I searched the specified directory and the "RDD.scala" file is present.

Looks like an issue with classpath. Check if it contains spark lib. — Sergii Lagutin, Jun 24 '16 at 08:38

score 0 · Answer 1 · edited May 23 '17 at 11:58

0

Based on the classPath you have mentioned I assume that your trying out the example from https://github.com/sryza/aas.

You haven't imported the project properly on your IDE. you can run the same code from the command line using maven.

#run this from the respective module's directory
mvn exec:java -Dexec.mainClass="com.cloudera.datascience.intro.RunIntro"

Note : While importing the project select on the parent pom.xml file. Intellij will resolve all the dependencies by itself.

Alternatively you can add the jars manually by following Correct way to add external jars

edited May 23 '17 at 11:58

Community

1
1

answered Jun 24 '16 at 16:55

Vishnu667

768
1
16
39

I tried it. Doesn't seem to be working. I'm getting the same error. – Saurav Sircar Jun 27 '16 at 07:32
Did you run this from the module's directory ? I've modified my answer i accidentally mentioned the parent directory. I have the same repo locally and its working. – Vishnu667 Jun 27 '16 at 07:39

score 0 · Answer 2 · answered Jun 27 '16 at 07:43

It might be the case that you have imported the project as a sbt project. You have to import the project as a maven project , and then run it !

You can do this by installing the Maven plugin and then re-import the project as a maven project and then run it. I hope that helps !

Apache Spark error

2 Answers2