0

I am unable to start worker node for Apache Spark 2.1.1, (I have single node setup, my Master and Worker will be running on same host), below are the logs:

$ ./start-all.sh

starting org.apache.spark.deploy.master.Master, logging to /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.master.Master-1-myhost.out localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out localhost: failed to launch: nice -n 0 /home/kedar/software/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://myhost:7077 localhost: at java.lang.ClassLoader.loadClass(ClassLoader.java:358) localhost: at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) localhost: full log in /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out

below are logs:

$ cat /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/launcher/Main : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:803) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:442) at java.net.URLClassLoader.access$100(URLClassLoader.java:64) at java.net.URLClassLoader$1.run(URLClassLoader.java:354) at java.net.URLClassLoader$1.run(URLClassLoader.java:348) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:347) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

Any help is appreciated! Thanks!

Kedar S. Dixit
  • 70
  • 10

1 Answers1

3

The issue was my java alternatives were pointing to wrong java version, below are the steps to resolve the issue on ubuntu, and as we could see the Spark Also started successfully:

kedar@myhost:/opt$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/opt/jdk1.8.0_72/bin/javac"
javac
kedar@myhost:/opt$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/opt/jdk1.8.0_72/bin/javac" 1
kedar@myhost:/opt$ ll javac
ls: cannot access javac: No such file or directory
kedar@myhost:/opt$ sudo update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
Nothing to configure.
kedar@myhost:/opt$ sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).

  Selection    Path                                         Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-7-openjdk-amd64/bin/javac   1071      auto mode
  1            /opt/jdk1.8.0_72/bin/javac                    1         manual mode
  2            /usr/lib/jvm/java-7-openjdk-amd64/bin/javac   1071      manual mode

Press enter to keep the current choice[*], or type selection number: 1       
update-alternatives: using /opt/jdk1.8.0_72/bin/javac to provide /usr/bin/javac (javac) in manual mode
kedar@myhost:/opt$ sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).

  Selection    Path                                         Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-7-openjdk-amd64/bin/javac   1071      auto mode
* 1            /opt/jdk1.8.0_72/bin/javac                    1         manual mode
  2            /usr/lib/jvm/java-7-openjdk-amd64/bin/javac   1071      manual mode

Press enter to keep the current choice[*], or type selection number: kedar@myhost:/opt$ 
kedar@myhost:/opt$ 
kedar@myhost:/opt$ 
kedar@myhost:/opt$ sudo update-alternatives --install "/usr/bin/java" "java" "/opt/jdk1.8.0_72/bin/java" 1
kedar@myhost:/opt$ ll javac
ls: cannot access javac: No such file or directory
kedar@myhost:/opt$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      auto mode
  1            /opt/jdk1.8.0_72/bin/java                        1         manual mode
  2            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      manual mode

Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /opt/jdk1.8.0_72/bin/java to provide /usr/bin/java (java) in manual mode
kedar@myhost:/opt$ cd
kedar@myhost:~$ cd $SPARK_HOME
kedar@myhost:~/software/spark$ ./sbin/stop-all.sh 
localhost: no org.apache.spark.deploy.worker.Worker to stop
stopping org.apache.spark.deploy.master.Master
kedar@myhost:~/software/spark$ ./sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.master.Master-1-myhost.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out
Kedar S. Dixit
  • 70
  • 10