sbt-assembly: deduplication found error

Question

I am not sure whether mergestrategy or exclude jars is the best option here. Any help with how do I proceed further with this error will be great!

[sameert@pzxdcc0151 approxstrmatch]$ sbt assembly
[info] Loading project definition from /apps/sameert/software/approxstrmatch/project
[info] Set current project to approxstrmatch (in build file:/apps/sameert/software/approxstrmatch/)
[info] Including from cache: scala-library.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Including from cache: curator-client-2.4.0.jar
[info] Including from cache: secondstring-20140729.jar
[info] Including from cache: slf4j-api-1.7.5.jar
[info] Including from cache: jsr305-1.3.9.jar
[info] Including from cache: jul-to-slf4j-1.7.5.jar
[info] Including from cache: jcl-over-slf4j-1.7.5.jar
[info] Including from cache: commons-digester-1.8.jar
[info] Including from cache: compress-lzf-1.0.0.jar
[info] Including from cache: commons-beanutils-1.7.0.jar
[info] Including from cache: zookeeper-3.4.5.jar
[info] Including from cache: slf4j-log4j12-1.7.5.jar
[info] Including from cache: commons-beanutils-core-1.8.0.jar
[info] Including from cache: commons-net-2.2.jar
[info] Including from cache: commons-el-1.0.jar
[info] Including from cache: log4j-1.2.17.jar
[info] Including from cache: scala-library.jar
[info] Including from cache: jline-0.9.94.jar
[info] Including from cache: snappy-java-1.0.5.jar
[info] Including from cache: hsqldb-1.8.0.10.jar
[info] Including from cache: chill_2.10-0.3.6.jar
[info] Including from cache: oro-2.0.8.jar
[info] Including from cache: chill-java-0.3.6.jar
[info] Including from cache: kryo-2.21.jar
[info] Including from cache: reflectasm-1.07-shaded.jar
[info] Including from cache: minlog-1.2.jar
[info] Including from cache: guava-14.0.1.jar
[info] Including from cache: jetty-plus-8.1.14.v20131031.jar
[info] Including from cache: javax.transaction-1.1.1.v201105210645.jar
[info] Including from cache: jackson-mapper-asl-1.8.8.jar
[info] Including from cache: jackson-core-asl-1.8.8.jar
[info] Including from cache: jetty-webapp-8.1.14.v20131031.jar
[info] Including from cache: curator-recipes-2.4.0.jar
[info] Including from cache: jetty-xml-8.1.14.v20131031.jar
[info] Including from cache: spark-core_2.10-1.0.0.jar
[info] Including from cache: objenesis-1.2.jar
[info] Including from cache: curator-framework-2.4.0.jar
[info] Including from cache: hadoop-client-1.0.4.jar
[info] Including from cache: jetty-util-8.1.14.v20131031.jar
[info] Including from cache: scalap-2.10.4.jar
[info] Including from cache: akka-remote_2.10-2.2.3-shaded-protobuf.jar
[info] Including from cache: jetty-servlet-8.1.14.v20131031.jar
[info] Including from cache: jetty-security-8.1.14.v20131031.jar
[info] Including from cache: jetty-server-8.1.14.v20131031.jar
[info] Including from cache: javax.servlet-3.0.0.v201112011016.jar
[info] Including from cache: jetty-continuation-8.1.14.v20131031.jar
[info] Including from cache: jetty-http-8.1.14.v20131031.jar
[info] Including from cache: jetty-io-8.1.14.v20131031.jar
[info] Including from cache: hadoop-core-1.0.4.jar
[info] Including from cache: jetty-jndi-8.1.14.v20131031.jar
[info] Including from cache: xmlenc-0.52.jar
[info] Including from cache: commons-codec-1.4.jar
[info] Including from cache: javax.mail.glassfish-1.4.1.v201005082020.jar
[info] Including from cache: javax.activation-1.1.0.v201105071233.jar
[info] Including from cache: commons-math-2.1.jar
[info] Including from cache: commons-lang3-3.3.2.jar
[info] Including from cache: commons-configuration-1.6.jar
[info] Including from cache: metrics-core-3.0.0.jar
[info] Including from cache: metrics-jvm-3.0.0.jar
[info] Including from cache: metrics-json-3.0.0.jar
[info] Including from cache: commons-collections-3.2.1.jar
[info] Including from cache: metrics-graphite-3.0.0.jar
[info] Including from cache: commons-lang-2.4.jar
[info] Including from cache: akka-actor_2.10-2.2.3-shaded-protobuf.jar
[info] Including from cache: config-1.0.2.jar
[info] Including from cache: tachyon-0.4.1-thrift.jar
[info] Including from cache: netty-3.6.6.Final.jar
[info] Including from cache: protobuf-java-2.4.1-shaded.jar
[info] Including from cache: uncommons-maths-1.2.2a.jar
[info] Including from cache: akka-slf4j_2.10-2.2.3-shaded-protobuf.jar
[info] Including from cache: json4s-jackson_2.10-3.2.6.jar
[info] Including from cache: json4s-core_2.10-3.2.6.jar
[info] Including from cache: ant-1.9.0.jar
[info] Including from cache: json4s-ast_2.10-3.2.6.jar
[info] Including from cache: ant-launcher-1.9.0.jar
[info] Including from cache: paranamer-2.6.jar
[info] Including from cache: commons-io-2.4.jar
[info] Including from cache: jackson-core-2.3.0.jar
[info] Including from cache: pyrolite-2.0.1.jar
[info] Including from cache: colt-1.2.0.jar
[info] Including from cache: concurrent-1.3.4.jar
[info] Including from cache: py4j-0.8.1.jar
[info] Including from cache: mesos-0.18.1-shaded-protobuf.jar
[info] Including from cache: scala-compiler.jar
[info] Including from cache: jets3t-0.7.1.jar
[info] Including from cache: commons-httpclient-3.1.jar
[info] Including from cache: netty-all-4.0.17.Final.jar
[info] Including from cache: stream-2.5.1.jar
[info] Including from cache: scala-reflect.jar
[info] Including from cache: jackson-databind-2.3.0.jar
[info] Including from cache: jackson-annotations-2.3.0.jar
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/DEPENDENCIES' with strategy 'discard'
[info] Assembly up to date: /apps/sameert/software/approxstrmatch/app/target/scala-2.10/app-assembly-0.1-SNAPSHOT.jar

// Here is where I start seeing errors:

java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
/home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
    at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
    at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
    at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
    at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
    at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
    at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
    at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
    at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
    at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
    at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
    at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
    at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
    at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
    at sbt.std.Transform$$anon$4.work(System.scala:64)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.Execute.work(Execute.scala:244)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

// Here are the error messages.

[error] (approxstrmatch/*:assembly) deduplicate: different file contents found in the following:
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.transaction/orbits/javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.mail.glassfish/orbits/javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
[error] /home/sameert/.ivy2/cache/org.eclipse.jetty.orbit/javax.activation/orbits/javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
[error] Total time: 4 s, completed Aug 5, 2014 9:53:06 AM

https://stackoverflow.com/questions/75955027 https://stackoverflow.com/questions/74800073 https://stackoverflow.com/questions/74879217 https://stackoverflow.com/questions/74809158 — Dmytro Mitin, Apr 08 '23 at 08:52

score 306 · Answer 1 · edited Sep 27 '22 at 11:09

306

Add the following code to your build.sbt file:

assemblyMergeStrategy in assembly := {
 case PathList("META-INF", _*) => MergeStrategy.discard
 case _                        => MergeStrategy.first
}

edited Sep 27 '22 at 11:09

Xavier Guihot

54,987
21
291
190

answered Aug 20 '16 at 20:38

Elesin Olalekan Fuad

3,235
1
14
10

11

Actually I think we should just overwrite merge strategy for META-INF leaving old strategies for the rest, so: ```assemblyMergeStrategy in assembly := { case PathList("META-INF", xs @ _*) => MergeStrategy.discard\n case x => val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x) }``` – note Jan 05 '17 at 10:39
5

You can refer to [sbt-assembly/Merge Strategy](https://github.com/sbt/sbt-assembly#merge-strategy) for more details – Minh Thai Sep 18 '17 at 10:49
1

Worked for me, never heard anyone about this. – thebluephantom Jul 30 '18 at 18:14
7

Works well, but a small explaination on what it does wouldn't hurt – Maldus Aug 07 '18 at 15:24
4

It was not working as it is in Play 2.5; So, I replaced `assemblyMergeStrategy` by `mergeStrategy` & it worked! – Tushar Walzade Nov 02 '18 at 10:18
1

@TusharWalzade `mergeStrategy` is also working for me. – David Boreham Mar 11 '21 at 03:59
The reason this fixes is what you are telling SBT's assembly plugin to do is to merge a bunch of jar files. What do you do if multiple jar files have a file at the same path? And what if they are different? Adding this rule to default to just picking the first will usually get your build to work, but it doesn't do this by default because this could cause a situation where the "Wrong" file is chosen and included in the fat JAR. Hence SBT requires you to explicitly define how you want to handle conflicts. – ianpojman Sep 22 '21 at 20:47

score 100 · Answer 2 · edited Oct 17 '16 at 08:50

100

Use the "provided" configuration, which will scope your dependent library.

For example:

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" % "provided"

If needed, read more at

https://github.com/sbt/sbt-assembly#excluding-jars-and-files

edited Oct 17 '16 at 08:50

Prasad Khode

6,602
11
44
59

answered Oct 02 '14 at 09:11

uris

5,993
3
25
24

1

Sorry, I had a "skip update" in a script, DOH. This did work and is significantly simpler than the other answer. – samthebest Oct 30 '14 at 21:42
10

using "provided" works for me as well. it will exclude that provided dependencies and keep the jar package light. However, if the I want to pack all the dependencies/jars, how to do that? – HappyCoding Feb 10 '16 at 03:58
1

The drawback of this approach is that if the point of using the assembly plugin is to build a fat jar with all dependencies, this particular dependency won't be included. – Igor Urisman Jan 31 '23 at 01:43

score 8 · Answer 3 · answered Aug 05 '14 at 20:18

import AssemblyKeys._

name := "approxstrmatch"

version := "1.0"

scalaVersion := "2.10.4"

// unmanagedJars in Compile += file("lib/secondstring-20140729.jar")

libraryDependencies+="org.apache.spark"%%"spark-core"%"1.0.0"

libraryDependencies ++= Seq(
    ("org.apache.spark"%%"spark-core"%"1.0.0").
    exclude("org.eclipse.jetty.orbit", "javax.servlet").
    exclude("org.eclipse.jetty.orbit", "javax.transaction").
    exclude("org.eclipse.jetty.orbit", "javax.mail").
     exclude("org.eclipse.jetty.orbit", "javax.activation").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-collections", "commons-collections").
    exclude("com.esotericsoftware.minlog", "minlog")
)


resolvers += "AkkaRepository" at "http://repo.akka.io/releases/"



 lazy val app = Project("approxstrmatch", file("approxstrmatch"),
    settings = buildSettings ++ assemblySettings ++ Seq(
    mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
    {
        case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
        case PathList("javax", "transaction", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "mail", xs @ _*)     => MergeStrategy.first
        case PathList("javax", "activation", xs @ _*)     => MergeStrategy.first
        case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
        case "application.conf" => MergeStrategy.concat
        case "unwanted.txt"     => MergeStrategy.discard
        case x => old(x)
        }
    })
  )


mainClass in assembly := Some("approxstrmatch.JaccardScore")
// jarName in assembly := "approstrmatch.jar"

hagai · Answer 4 · 2022-04-20T12:40:20.387

6

The module-info.class file has moved in many libraries. Here is the updated solution

ThisBuild / assemblyMergeStrategy  := {
  case PathList("module-info.class") => MergeStrategy.discard
  case x if x.endsWith("/module-info.class") => MergeStrategy.discard
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

edited Apr 20 '22 at 12:40

answered Jun 11 '21 at 13:13

hagai

424
1
7
13

sbt-assembly: deduplication found error

4 Answers4

Linked