ClassNotFoundException in submitting a jar using spark-submit

Question

I already checked the other topics but I didn't find a good solution, I am working with intelliJ Idea and I created an SBT project.

Here's my code :

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object mdr {
  def main(args: Array[String]) {
    val logFile = "C:/Users/me/README.md"
    val jarfile = "file:///C:/Users/me/IdeaProjects/untitled/out/artifacts/untitled_jar/untitled.jar"
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]").setJars(Array(jarfile))
    conf.set("spark.eventLog.enabled", "true")
    conf.set("spark.eventLog.dir", "file:///C:/Users/me/spark/logs")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
      val numAs = logData.filter(line => line.contains("a")).count()
      val numBs = logData.filter(line => line.contains("b")).count()
      println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }    
}

Here's my command line :

spark-submit --driver-class-path C:\Users\me\IdeaProjects\untitled\out\artifacts\untitled_jar\untitled.jar --class mdr  --master local[*]  C:\Users\me\IdeaProjects\untitled\out\artifacts\untitled_jar\untitled.jar

And here's my error :

java.lang.ClassNotFoundException: mdr
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Unknown Source)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:700)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

try spark-submit --class your.class --master spark://yourSparkMasterHostname:7077 /provide.jar — Nayan Sharma, Jul 20 '17 at 08:32
I have a doubt, what is SparkMasterHostname please ? EDIT : I have a proxy so I'm not sure about it — Tiffany, Jul 20 '17 at 08:46
First of all: Please use upper case/came case object names: "Mdr". Secondly: is mdr.class contained in the jar file and in the correct package? — Rick Moritz, Jul 20 '17 at 11:17
Ok, I created a brand new project with a class Main, still get the same error. I'm going to check if the class is in the jar file now, maybe intelliJ is not doing what I want — Tiffany, Jul 20 '17 at 12:02
The Main class can't be found indeed, I think it is an IntelliJ issue then — Tiffany, Jul 20 '17 at 12:12

score -1 · Accepted Answer · answered Jul 20 '17 at 12:36

-1

Thanks to Rick Moritz I figured out that IntelliJ didn't make the artifacts as expected, the top answer of this topic solved the issue : How do I create a runnable JAR in IntelliJ as I would in Eclipse

answered Jul 20 '17 at 12:36

Tiffany

263
1
13

ClassNotFoundException in submitting a jar using spark-submit

1 Answers1