4

I want to create a big Jar file. for which am trying to use SBT ASSEMBLY. I installed sbt-assembly from GitHub and this answer. When I ran sbt assembly, I got this error:

java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/UserName/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-2.5.0.v201103041518.jar:javax/servlet/SingleThreadModel.class
/home/UserName/.ivy2/cache/org.mortbay.jetty/servlet-api/jars/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class

To solve this, I followed User's README page and this is the code he suggests. 

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "apache", xs @ _*) => MergeStrategy.last
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case PathList("project.clj") => MergeStrategy.last
    case PathList("overview.html") => MergeStrategy.last
    case x => old(x)
  }
}

Even after adding this code, I get the same error mentioned before. Please let me know what I am missing. Any help on this error will be appreciated. Thanks!

Update 2 :

Added the exlusion rule as per the link given, 

libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "0.8.0-incubating","com.codahale" % "jerkson_2.9.1" % "0.5.0","org.skife.com.typesafe.config" % "typesafe-config" % "0.3.0").map(_.exclude("javax", "servlet"))

Update 3:

I can locate the library that is causing the issue. 

| +-org.apache.avro:avro-ipc:1.7.4
| | +-io.netty:netty:3.4.0.Final (evicted by: 3.5.4.Final)
| | +-io.netty:netty:3.5.4.Final
...
...
| | +-org.mortbay.jetty:jetty-util:6.1.26
| | +-org.mortbay.jetty:jetty:6.1.26
| | | +-org.mortbay.jetty:jetty-util:6.1.26
| | | +-org.mortbay.jetty:servlet-api:2.5-20081211
| | | 
| | +-org.mortbay.jetty:servlet-api:2.5-20081211
| | +-org.slf4j:slf4j-api:1.7.2
...
... 
| +-org.eclipse.jetty:jetty-server:7.6.8.v20121106
| | +-org.eclipse.jetty.orbit:javax.servlet:2.5.0.v201103041518

Update 4 : Fix

So adding MergeStrategy did solve the problem. Even though I had quite a few dependencies, more than 10, adding MergeStrategy for each one of them individually solved the issue.

Community
  • 1
  • 1
Learner
  • 1,685
  • 6
  • 30
  • 42

2 Answers2

4

I think you are trying to treat the symptoms, but your problem actually isn't assembly: Your project has a library twice in two different versions on the class path (javax.servlet).

If these versions are binary compatible (which I don't know), you would be fine by excluding one of the two occurrences in your build file like so. If they incompatible, you will need to unroll your dependency graph (a good way to do this might be the sbt-dependency-graph plugin) and try to find matching versions.

In any case, it might be useful to (at least transitorily) keep the libraries in the project folder. If you add retrieveManaged in ThisBuild := true to your sbt build file, all the libraries will be found in <project-root>/lib_managed. This allows you to see which jars are actually there.

EDIT: Showing the dependency graph:

Add to project/plugins.sbt:

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")

Add to build.sbt:

net.virtualvoid.sbt.graph.Plugin.graphSettings

Then run sbt dependency-graph.

Community
  • 1
  • 1
0__
  • 66,707
  • 21
  • 171
  • 266
  • Thanks! Actually, I have already added `retrieveManaged` argument in sbt file.I can see the jar the that is causing this issue under `lib_managed/jars/org.mortbay.jetty/servlet-api/`. As per the link, I added [Update 2,in question] and still I face the issue. How to locate the exact library that is causing the issue ? – Learner Oct 26 '13 at 20:41
  • 1
    Have you used the dependency graph plugin? That should help you find out where the jar comes from. I have added the information. – 0__ Oct 26 '13 at 20:51
  • I have got the above dependency tree.Any pointers on how do I workaround this MergeStrategy error ? – Learner Oct 26 '13 at 23:39
  • 1
    I don't see `javax.servlet-2.5.0.v201103041518.jar` in your print out. Anyway, if you find the artefact from which it is referenced, you can add an `exclude` statement there, hoping that you won't run into a binary class incompatibility. If you _do_ (e.g. there are `AbstractMethodError`s), the only solution is to find versions of your dependencies which have a matching version for that library. If all fails, you might have to fork one of those projects and compile it against the servlet-api version you are using... – 0__ Oct 27 '13 at 00:20
  • oops, my bad. I missed that `javax.servlet-2.5.0.v201103041518.jar`! Updated the dependency graph output. – Learner Oct 27 '13 at 01:00
4

You can also refer This Tech Blog

Creating a single JAR for Spark project using sbt-assembly

This post is about how to create a fat jar for spark streaming project using sbt plugin. sbt-assembly is a sbt plugin to create a fat JAR of sbt project with all of its dependencies.

USB
  • 6,019
  • 15
  • 62
  • 93