ClassNotFoundException thrown launching Spark Shell

Question

I am trying to launch the spark shell for python from the directory using ./bin/pyspark

When i run the command i get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError:    org/apache/spark/launcher/Main
Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

It appears that I am missing: org.apache.spark.launcher.main

I am not quite sure how to resolve this issue and was wondering if anyone had any suggestions or had run into similar issues.

Thanks

A java error thrown by a java library in java code. Why is this tagged 'python'? — RobertB, Oct 12 '15 at 18:00
@RobertB pyspark is a python REPL driver on top of a java library/framework/whatever. Question is not completely detached from python. — mehmetminanc, Oct 12 '15 at 18:12
what version of spark are you trying to run? If you have no preference, and just want to get things running i would recommend "Pre-Built for Hadoop 2.6 and later" http://spark.apache.org/downloads.html — J'e, Oct 12 '15 at 18:29
And the package type I downloaded was the "pre-built for Hadoop 2.6 and later" — Padraig Barry, Oct 13 '15 at 15:29

score 8 · Answer 1 · edited May 23 '17 at 11:46

8

I also ran into this problem (on my Mac).

I followed the steps in this page: https://stackoverflow.com/a/14875241/5478610 and was able to get past this error.

I think the problem is that even though I had installed Java 8 on my Mac, when I ran from the command line it was still calling Java 6.

The jar file with the class (in ./lib/spark-assembly-1.5.1-hadoop2.6.0.jar) couldn't be opened with Java 6. But once I updated my the links so that calling java from the terminal would use Java 8, I was able to bring up pyspark.

edited May 23 '17 at 11:46

Community

1
1

answered Oct 23 '15 at 07:14

David Wen

81
4

Seconding that newer versions of Spark don't appear to support Java 6. I had the same issue on Spark 1.5.2, and upgrading to Java 7 resolved it. – Hammer Bro. Nov 24 '15 at 01:34
I know this thread is addressing the newer versions of Spark - but this was exactly my issue running an older version (1.6) with Java 6. My working setup: Hadoop:1.2.1, Spark 1.6, Java 7u79 – miniscem Aug 25 '16 at 20:37

angerhang · Answer 2 · 2015-12-08T23:46:31.470

In addition to David's answer, use

# Change java version to 1.7
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)

# Change java version to 1.8
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

to change the java version if you have multiple java versions installed and want to switch between them.

I also solved this problem by changing Java version to 1.8. The error message is not really pertinent in this case.

ClassNotFoundException thrown launching Spark Shell

2 Answers2