1

I am executing the java archive on Spark using spark-submit in Ubuntu. The command is given below. This JAR file was build using Maven Package. The dependencies are specified in pom.xml file.

]$ spark-submit --class HighScore.Driver --master local[*] JarfilePath/Levelwise_PCFS-0.0.1-SNAPSHOT.jar InputFilePath/K9_Site1.csv 1000.

I am getting following error even when packageName.className (HighScore.Driver) is specified in the command.

Here is the error message.

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/lwhite1/tablesaw/api/Table
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:727)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: com.github.lwhite1.tablesaw.api.Table
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 10 more

com/github/lwhite1/tablesaw/api/Table dependency was also specified in pom.xml file. But still it throws the exception.

Can some one help me in rectifying this error.

HDJEMAI
  • 9,436
  • 46
  • 67
  • 93
Lokeswari Venkataramana
  • 73
  • 1
  • 1
  • 7

1 Answers1

0

Remember that the classloader (specifically java.net.URLClassLoader) will look for classes in package a.b.c in folder a/b/c/ in each entry in your classpath.

NoClassDefFoundError can also indicate that you're missing a transitive dependency of a .jar file that you've compiled against and you're trying to use.

For example, if you had a class com.example.Foo, after compiling you would have a class file Foo.class. Say for example your working directory is .../project/. That class file must be placed in .../project/com/example, and you would set your classpath to .../project/.

Please refer this post for more details. It would be helpful.

coderpc
  • 4,119
  • 6
  • 51
  • 93
  • The .class file is present in /home/workspace/project/target/classes/HighScore/Driver.class . The classpath file has . I hope classpath is set correctly. I could not figure out where it is going wrong. – Lokeswari Venkataramana Jun 18 '17 at 11:30
  • I have two classes under HighScore Package. One is HighScore.Driver which is the having main class, and another one is HighScore.AttributeSelector called from Driver class. Do i need to specify both classes in above spark-submit command as ]$spark-submit --class HighScore.Driver, HighScore.AttributeSelector --master local[*] /home/hadoopuser/workspace/Levelwise_PCFS/target/Levelwise_PCFS-0.0.1-SNAPSHOT.jar InputFilePath/K9_Site1.csv 1000. Tried this too. This still says ClassNotFoundException. – Lokeswari Venkataramana Jun 18 '17 at 12:32