ERROR when using SPARK : java.lang.NoClassDefFoundError: dispatch/Http

Question

When using spark-submit this error always happen:

ERROR Executor: Exception in task 0.0 in stage 137.0 (TID 35)
java.lang.NoClassDefFoundError: dispatch/Http$

although I have import the lib (I'm using scala) :

import dispatch._, Defaults._

Part of code :

def publishToNsq(data: String) = {
        val topic = "post_similar_items_to_elasticsearch"
        val host = "192.168.100.160:4151"
        def myRequest = url(s"$host/pub?topic=$topic")
        def myRequestAsJson = myRequest.setContentType("application/json", "UTF-8")
        def myPostWithBody = myRequestAsJson << data
        val response = Http(myPostWithBody OK as.String)
        println(response)
    }

This is part of my build.sbt :

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
libraryDependencies += "org.json4s" %% "json4s-native" % "3.4.0"
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.4.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "1.1.4"
libraryDependencies += "net.databinder.dispatch" %% "dispatch-core" % "0.11.2"
resolvers += Resolver.mavenLocal

Any solution ?

That's because you need to assembly your application creating an uber jar. — eliasah, Jun 28 '16 at 11:57

score 0 · Accepted Answer · edited May 23 '17 at 12:16

0

The dependency is not in the jar, it would be resolved if you configure as in java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils

edited May 23 '17 at 12:16

Community

1
1

answered Jun 28 '16 at 11:57

yanghaogn

833
7
15

ERROR when using SPARK : java.lang.NoClassDefFoundError: dispatch/Http

1 Answers1