Scala: SBT: mergeStrategy: deduplicate: different file contents found in the following:

Question

I am trying to build a fat Jar using the sbt-assembly plugin. I am on SBT version 0.13.1 and assembly plugin version: 0.11.2. Whenever I run the assembly command in the sbt console I am encountering this error:

Update:

    [warn] Merging 'META-INF\LICENSE' with strategy 'rename'
    [warn] Merging 'META-INF\DEPENDENCIES' with strategy 'discard'
    [warn] Merging 'META-INF\MANIFEST.MF' with strategy 'discard'
    [debug] Merging 'META-INF\native\linux32\libleveldbjni.so' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\linux64\libleveldbjni.so' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\osx\libleveldbjni.jnilib' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\windows32\leveldbjni.dll' with strategy 'deduplicate'
    [debug] Merging 'META-INF\native\windows64\leveldbjni.dll' with strategy 'deduplicate'
    [debug] Merging 'default.key' with strategy 'deduplicate'
    java.lang.RuntimeException: deduplicate: different file contents found in the following:
    mydef.key
    mydef.key
            at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
            at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
            at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
            at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
            at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
            at scala.collection.Iterator$class.foreach(Iterator.scala:727)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
            at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
            at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
            at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
            at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
            at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
            at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
            at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
            at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
            at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
            at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
            at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
            at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
            at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
            at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
            at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
            at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
            at sbt.std.Transform$$anon$4.work(System.scala:64)
            at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
            at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
            at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
            at sbt.Execute.work(Execute.scala:244)
            at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
            at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
            at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
            at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
    [error] (cloud_ui/*:assembly) deduplicate: different file contents found in the following:
    [error] mydef.key
    [error] mydef.key

**[error](mymodule:*.assembly) deduplicate: different file contents found in the following:
[error] mydef.key
[error] mydef.key**

On inspecting my build.sbt I find that IntelliJ complains about "mergeStrategy in assembly" block. Its complaint is: Type mismatch expected: Def Initialize[(String) => Plugin.MergeStrategy], actual: DefInitialize[Any]

Here is my mergeStrategy:

       mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
             {
                case PathList("org", "jboss", "netty", xs @ _*)  =>  MergeStrategy.first
      case PathList("META-INF", "jboss-beans.xml")         => MergeStrategy.first
      case PathList("org", "w3c", "dom", "TypeInfo.class")  => MergeStrategy.first
      case "jboss-beans.xml" => MergeStrategy.first
      case "application.conf" => MergeStrategy.concat
      case "unwanted.txt"     => MergeStrategy.discard
      case "rootdoc.txt"     => MergeStrategy.discard
      case x => old(x)
    }
   }

I have looked up some of the resources on stackoverflow that address a similar problem and I am wondering if the following resource has the solution to my problem.

From a knowledge perspective, I have learned just enough to use SBT for my build purposes, but I have not got been able to delve into its murky depths so far. But I am ready to work on it. If someone can point me in the right direction I can go off and correct the error in the build.sbt file and also my 'deduplicate error"so that I can accomplish my goal of building a fat jar that I can deploy..

Update on what I tried myself, but failed to get it work: I had modified my mergeStrategy to include this line:

case PathList(ps @ _*) if ps.last endsWith ".key" => MergeStrategy.first

Modified mergeStrategy now looks like:

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => 
 {
  case PathList(ps @ _*) if ps.last endsWith ".key" => MergeStrategy.first
  case PathList("org", "jboss", "netty", xs @ _*)         => MergeStrategy.first
  case PathList("META-INF", "jboss-beans.xml")         => MergeStrategy.first
  case PathList("org", "w3c", "dom", "TypeInfo.class")         => MergeStrategy.first
  case "jboss-beans.xml" => MergeStrategy.first
  case "application.conf" => MergeStrategy.concat
  case "unwanted.txt"     => MergeStrategy.discard
  case "rootdoc.txt"     => MergeStrategy.discard
  case x =>
    //val old = (mergeStrategy in assembly).value
       old(x)
   }
}

Update: I solved the problem. I simply deleted the extra files and rebuilt again. And I did not have to do anything else to the merge-strategy.

A minimal, reproducible build would help this question find its answer. — Dale Wijnand, Mar 19 '15 at 17:42

score 4 · Answer 1 · answered Mar 19 '15 at 18:37

4

It looks like you got 2 files called mydef.key But because their content is not the same and sbt-assembly doesn't have rule how to merge it then exception is raised.

Check what are both contents and if they are important for your build. If not just try to add:

case "mydef.key" => MergeStrategy.first

answered Mar 19 '15 at 18:37

kpbochenek

469
2
21

I tried your suggestion, but it complained again with the same error. – user3825558 Mar 19 '15 at 19:33
can you actually verify you got 2 files called mydef.key? If so can you tell us what is insied? Just for debugging can you modify "case x => " to look like "case x => println(x)" and show us what last x was? – kpbochenek Mar 19 '15 at 20:14
Thanks. I will do that and let you know. – user3825558 Mar 19 '15 at 21:50
Yes, there are actually two files – user3825558 Mar 19 '15 at 21:50
Can you actually verify your rule grabs mydef.key? Like adding println statement in your "case ... => println(ps) MergeStrategy.first" And even better can you remove one of mydef.key files to verify that caused your issue? – kpbochenek Mar 19 '15 at 22:10
For now, I have solved my problem by manually deleting the extra files. Thanks though,. – user3825558 Mar 23 '15 at 04:12
It says your duplicated file is "mydef.key". This is full path which means it is in a root directory. PathList when you look at implementation expects path with a directory and then a file. But because your file is in a root directory your rule doesn't match. This is what I think is happening. – kpbochenek Mar 23 '15 at 08:15

Scala: SBT: mergeStrategy: deduplicate: different file contents found in the following:

1 Answers1