spark compilation fails with sbt-assembly 0.13.8

Question

I'm trying to build a spark app after upgrading sbt-assembly to 0.13.8. This is the content of the build.sbt that i have right now. But this doesn't work. It seems to be a problem with the default mergeStrategy. I'am getting around 600+ deduplicate errors

name := "volumeApp"

version := "0.0.1"

scalaVersion := "2.10.4" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-streaming-kafka" % "1.3.1",
    "org.apache.spark" %% "spark-core" % "1.3.1",
    "org.apache.spark" %% "spark-streaming" % "1.3.1",
    "org.apache.kafka" %% "kafka" % "0.8.2.1",
    "com.datastax.spark" %% "spark-cassandra-connector" % "1.3.0-M1",
    "com.github.vonnagy" %% "service-container-metrics-reporting" % "1.0.1" exclude("com.codahale.metrics", "metrics-core"),
    "joda-time" % "joda-time" % "2.7",
    "log4j" % "log4j" % "1.2.14"
)

assemblyJarName in assembly := "inventoryVolume.jar"

assemblyMergeStrategy in assembly := {
    case PathList("META-INF", "jboss-beans.xml") => MergeStrategy.first
    case PathList("META-INF", "mailcap") => MergeStrategy.discard
    case PathList("META-INF", "maven", "org.slf4j", "slf4j-api", xa @ _*) => MergeStrategy.rename
    case PathList("META-INF", "ECLIPSEF.RSA") => MergeStrategy.discard
    case PathList("META-INF", "mimetypes.default") => MergeStrategy.first
    case PathList("com", "datastax", "driver", "core", "Driver.properties") => MergeStrategy.last
    case PathList("com", "esotericsoftware", "minlog", xx @ _*) => MergeStrategy.first
    case PathList("plugin.properties") => MergeStrategy.discard
    case PathList("javax", "activation", xw @ _*) => MergeStrategy.first
    case PathList("org", "apache", "hadoop", "yarn", xv @ _*) => MergeStrategy.first
    case PathList("org", "apache", "commons", xz @ _*) => MergeStrategy.first
    case PathList("org", "jboss", "netty", ya @ _*) => MergeStrategy.first
    case x =>
        val baseStrategy = (assemblyMergeStrategy in assembly).value
        baseStrategy(x)
}

I have attached the errors over here http://pastebin.com/T6HRJ6Kv Can someone help me out .. I am completely new to build tools . I tried reading the sbt guide but was lost in no time.

http://stackoverflow.com/questions/30414782/proper-way-to-make-a-spark-fat-jar-using-sbt?rq=1 — sainath reddy, Jul 04 '15 at 15:15

score 0 · Accepted Answer · answered Jul 04 '15 at 23:43

0

It seems that you are trying to build an assembly jar which includes multiple versions of the akka actor system. There are a few possible solutions, one would be to exclude it from your dependencies (see http://www.scala-sbt.org/0.13/docs/Library-Management.html for an idea) or alter your merge strategy but I'd recommend the excluding dependency approach.

answered Jul 04 '15 at 23:43

Holden

7,392
1
27
33

Thanks for pointing that out @Holden While excluding dependency on akka_actor i was not able to figure out the syntax for that .. one of the lines in my build.sbt already has one exclude statements didn't have any effect as i was still getting deduplicate errors w.r.t metrics-core. What worked out for me was a default case in merge strategy with MergeStrategy.first. I will try to figure out a better solution to this later . Whats strange though is i didnot have any issues when i was using sbt-assembly 0.11.+ – sainath reddy Jul 06 '15 at 03:24

spark compilation fails with sbt-assembly 0.13.8

1 Answers1