Spark submit ClassNotFoundException

Question

This problem is old as Apache Spark itself, but still neither of solutions helped me. I have following class declaration:

object Main {
  def main(args: Array[String]): Unit = {
  ...
}

Class reference is org.griat.rcse.Main (ctrl+shift+alt+c in IntelliJ), the same stays in pom.xml:

<groupId>org.griat.rcse</groupId>

The way I try to run it:

spark-submit --master yarn --deploy-mode client --class org.griat.rcse.Main Glonass112-1.0-SNAPSHOT.jar

And once it has worked, but caught an exception in code. After I fixed it, I cleaned and packaged it. Though pom.xml didn't change (as well as paths and class names), it stopped submitting to Spark, showing ClassNotFoundException:

java.lang.ClassNotFoundException: org.griat.rcse.Main
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:810)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I also added maven-jar-plugin, not that it would help:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
                <archive>
                    <manifest>
                        <addClasspath>true</addClasspath>
                        <mainClass>org.griat.rcse.Main</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>

What is the source of this black magic and how can I handle it?

Make sure the Main class is in correct directory, or you can try with other plugins like `Maven Assembly Plugin`, can you also add all maven content and your location of Main class from root directory of project ? — koiralo, Aug 05 '20 at 05:01

score 1 · Accepted Answer · answered Aug 05 '20 at 20:51

It seems you haven't created a "fat jar", a jar with all needed dependencies. When you do not do this, or if some dependencies are "provided", those dependencies would otherwise need to be in the overall class path (check environment variables ending in CLASSPATH.

To create a fat jar, you need to use the Assembly plugin, as noted. Please see the question:

What are the differences between Maven Jar Plugin and Maven Assembly Plugin?

Here is an example of a pom file with the assembly plugin added.

Relevant lines:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.3</version>
</plugin>

Here are overall instructions for the assembly plugin -- note that you may want to use a descriptor XML file, explained on the bottom of the page.

You can also easily confirm if your jar contains the class: the class should be listed when you run jar tf path/to/assembled.jar, as documented. You can use grep to make it easier: jar tf path/to/assembled.jar|grep org.griat.rcse

Hmm, seemes, like maven doesn't put any classes to the .jar file and it doesn't depend on this plugin. However, it seemes to be topic for another question. — Arli Chokoev, Aug 08 '20 at 10:48
By the way, manual classes compilation and then running `package` helps. — Arli Chokoev, Aug 08 '20 at 11:15
The pom file must have `maven-assembly-plugin` in it with a configuration, for the plugin to execute when `package` is run. Here is [another example](https://mkyong.com/maven/create-a-fat-jar-file-maven-assembly-plugin/) — ELinda, Aug 08 '20 at 16:31

Spark submit ClassNotFoundException

1 Answers1