JodaTime issues with scala and spark when invoking spark-submit

Question

I am having trouble using JodaTime in a spark scala program. I tried the solutions posted in the past in Stackoverflow and they don't seem to fix the issue for me.

When I try to spark-submit it comes back with an error like the following:

   15/09/04 17:51:57 INFO Remoting: Remoting started; listening on addresses :        
                     [akka.tcp://sparkDriver@100.100.100.81:56672]
   Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/DateTimeZone
      at com.ttams.xrkqz.GenerateCsv$.main(GenerateCsv.scala:50)
      at com.ttams.xrkqz.GenerateCsv.main(GenerateCsv.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

After sbt package, which seems to work fine, I invoke spark-submit like this ... ~/spark/bin/spark-submit --class "com.ttams.xrkqz.GenerateCsv" --master local target/scala-2.10/scala-xrkqz_2.10-1.0.jar

In my build.sbt file, I have

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"

libraryDependencies ++= Seq ("joda-time" % "joda-time" % "2.8.2",
                         "org.joda" % "joda-convert" % "1.7"
                       )

I have tried multiple versions of joda-time and joda-convert but am not able to use spark-submit from the command line. However, it seems to work when i run within the ide (scalaide).

Let me know if you have any suggestions or ideas.

Answer 1

It seems that you are missing the dependencies in your class path. You could do a few things: One is you could manually add the joda time jars to spark submit with the --jars argument, the other is you could use the assembly plugin and build an assembly jar (you will likely want to mark spark-core as "provided" so it doesn't end up in your assembly) which will have all of your dependencies.