assembly-merge-strategy issues using sbt-assembly

Question

I am trying to convert a scala project into a deployable fat jar using sbt-assembly. When I run my assembly task in sbt I am getting the following error:

Merging 'org/apache/commons/logging/impl/SimpleLog.class' with strategy 'deduplicate'
    :assembly: deduplicate: different file contents found in the following:
    [error] /Users/home/.ivy2/cache/commons-logging/commons-logging/jars/commons-logging-1.1.1.jar:org/apache/commons/logging/impl/SimpleLog.class
    [error] /Users/home/.ivy2/cache/org.slf4j/jcl-over-slf4j/jars/jcl-over-slf4j-1.6.4.jar:org/apache/commons/logging/impl/SimpleLog.class

Now from the sbt-assembly documentation:

If multiple files share the same relative path (e.g. a resource named application.conf in multiple dependency JARs), the default strategy is to verify that all candidates have the same contents and error out otherwise. This behavior can be configured on a per-path basis using either one of the following built-in strategies or writing a custom one:

MergeStrategy.deduplicate is the default described above

MergeStrategy.first picks the first of the matching files in classpath order

MergeStrategy.last picks the last one

MergeStrategy.singleOrError bails out with an error message on conflict

MergeStrategy.concat simply concatenates all matching files and includes the result

MergeStrategy.filterDistinctLines also concatenates, but leaves out duplicates along the way

MergeStrategy.rename renames the files originating from jar files

MergeStrategy.discard simply discards matching files

Going by this I setup my build.sbt as follows:

import sbt._
import Keys._
import sbtassembly.Plugin._
import AssemblyKeys._
name := "my-project"
version := "0.1"
scalaVersion := "2.9.2"
crossScalaVersions := Seq("2.9.1","2.9.2")

//assemblySettings
seq(assemblySettings: _*)

resolvers ++= Seq(
    "Typesafe Releases Repository" at "http://repo.typesafe.com/typesafe/releases/",
    "Typesafe Snapshots Repository" at "http://repo.typesafe.com/typesafe/snapshots/",
    "Sonatype Repository" at "http://oss.sonatype.org/content/repositories/releases/"
)

libraryDependencies ++= Seq(
    "org.scalatest" %% "scalatest" % "1.6.1" % "test",
    "org.clapper" %% "grizzled-slf4j" % "0.6.10",
    "org.scalaz" % "scalaz-core_2.9.2" % "7.0.0-M7",
    "net.databinder.dispatch" %% "dispatch-core" % "0.9.5"
)

scalacOptions += "-deprecation"
mainClass in assembly := Some("com.my.main.class")
test in assembly := {}
mergeStrategy in assembly := mergeStrategy.first

In the last line of the build.sbt, I have:

mergeStrategy in assembly := mergeStrategy.first

Now, when I run SBT, I get the following error:

error: value first is not a member of sbt.SettingKey[String => sbtassembly.Plugin.MergeStrategy]
    mergeStrategy in assembly := mergeStrategy.first

Can somebody point out what I might be doing wrong here?

Thanks

`mergeStrategy` is deprecated now. Try instead `assemblyMergeStrategy`. I'm using assembly version `0.14.10` — luca.giovagnoli, Feb 05 '20 at 00:41

score 15 · Answer 1 · answered Mar 25 '14 at 15:28

As for the current version 0.11.2 (2014-03-25), the way to define the merge strategy is different.

This is documented here, the relevant part is:

NOTE: mergeStrategy in assembly expects a function, you can't do
mergeStrategy in assembly := MergeStrategy.first

The new way is (copied from the same source):

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
    case "application.conf" => MergeStrategy.concat
    case "unwanted.txt"     => MergeStrategy.discard
    case x => old(x)
  }
}

This is possibly applicable to earlier versions as well, I don't know exactly when it has changed.

The last example (which is official) doesn't work on scala 10, showing Type mismatch — tribbloid, Jul 20 '14 at 08:51
@tribbloid I have tested the snippet in an `assembly.sbt`, and it works (Scala 2.10.3, sbt 0.13.2). — Beryllium, Jul 22 '14 at 08:10

score 13 · Accepted Answer · edited Oct 26 '13 at 12:28

13

I think it should be MergeStrategy.first with a capital M, so mergeStrategy in assembly := MergeStrategy.first.

edited Oct 26 '13 at 12:28

Seth Tisue

29,985
11
82
149

answered Feb 10 '13 at 11:46

Slapping my head. That was such an oversight. Thanks for pointing it out. – sc_ray Feb 11 '13 at 07:21
1

doesn't work for me, it says: build.sbt:27: error: not found: value mergeStrategy mergeStrategy in assembly := MergeStrategy.first ^ [error] Type error in expression – stantonk Feb 28 '14 at 21:44
@gjain no, i had to basically copy the entire code and change how the mergeStrategy resolves collisions by accepting the first version instead of failing out (aka what maven does): https://gist.github.com/stantonk/5303b7ec84b782a58628 I don't know why sbt doesnt do this... – stantonk Jun 12 '14 at 04:18
2

This is not correct. according to the documentation. You can't do `mergeStrategy in assembly := MergeStrategy.first` because `mergeStrategy in assembly` expects a function and you'll get a `type mismatch error` – eliasah Jul 21 '14 at 12:09

score 12 · Answer 3 · answered Jan 04 '17 at 22:06

this is the proper way to merge most of the common java/scala projects. it takes care of META-INF and classes.

also the service registration in META-INF is taken care of.

assemblyMergeStrategy in assembly := {
case x if Assembly.isConfigFile(x) =>
  MergeStrategy.concat
case PathList(ps @ _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
  MergeStrategy.rename
case PathList("META-INF", xs @ _*) =>
  (xs map {_.toLowerCase}) match {
    case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
      MergeStrategy.discard
    case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
      MergeStrategy.discard
    case "plexus" :: xs =>
      MergeStrategy.discard
    case "services" :: xs =>
      MergeStrategy.filterDistinctLines
    case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
      MergeStrategy.filterDistinctLines
    case _ => MergeStrategy.first
  }
case _ => MergeStrategy.first}

+1 for the META-INF rules. That helped me fix an issue with packaging jpmml-spark together with other libs into one jar. — Elmar Macek, Mar 10 '21 at 15:01
Agreed with Elmar – thank you! The accepted answer more directly addresses OP's issue, but these META-INF rules are gold; most suggestions (on this page and elsewhere) discard everything under META-INF but this loses needed config for many libraries. — AlterEgo, Aug 05 '22 at 14:21

score 8 · Answer 4 · answered Jul 04 '15 at 09:21

I have just setup a little sbt project that needs to rewire some mergeStrategies, and found the answer a little outdated, let me add my working code for versions (as of 4-7-2015)

sbt 0.13.8
scala 2.11.6

assembly 0.13.0

mergeStrategy in assembly := {
  case x if x.startsWith("META-INF") => MergeStrategy.discard // Bumf
  case x if x.endsWith(".html") => MergeStrategy.discard // More bumf
  case x if x.contains("slf4j-api") => MergeStrategy.last
  case x if x.contains("org/cyberneko/html") => MergeStrategy.first
  case PathList("com", "esotericsoftware", xs@_ *) => MergeStrategy.last // For Log$Logger.class
  case x =>
     val oldStrategy = (mergeStrategy in assembly).value
     oldStrategy(x)
}

`mergeStrategy` is deprecated now, use `assemblyMergeStrategy` instead. — Chunliang Lyu, Aug 18 '15 at 14:35

score 4 · Answer 5 · edited May 23 '17 at 11:46

For the new sbt version (sbt-version :0.13.11), I was getting the error for slf4j; for the time being took the easy way out : Please also check the answer here Scala SBT Assembly cannot merge due to de-duplication error in StaticLoggerBinder.class where sbt-dependency-graph tool is mentioned which is pretty cool to do this manually

assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) {
  (old) => {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
}

score 1 · Answer 6 · answered Feb 22 '19 at 18:01

1

Quick update: mergeStrategy is deprecated. Use assemblyMergeStrategy. Apart from that, earlier responses are still solid

answered Feb 22 '19 at 18:01

Jake

4,322
6
39
83

score 0 · Answer 7 · answered Sep 20 '19 at 07:48

Add following to build.sbt to add kafka as source or destination

 assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 //To add Kafka as source
 case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => 
 MergeStrategy.concat
 case x => MergeStrategy.first
 }

assembly-merge-strategy issues using sbt-assembly

7 Answers7

Linked