sbt assembly command shows error: (redshiftConnector/*:assembly) deduplicate: different file contents found

Question

sbt assembly is giving me errors i am trying to create a fat jar with this link https://github.com/sbt/sbt-assembly under the heading Exclude specific transitive deps they exclude some of the libs that are causing this issue but in my projects build.sbt file no dependency is there which is showing on sbt console . here is the stack trace

[info] Including from cache: scala-library-2.11.7.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] Assembly up to date: /home/user/Downloads/ScalaProj/datasembly/target/scala-2.11/datasembly-assembly-0.1-SNAPSHOT.jar
[info] Including from cache: jackson-datatype-jsr310-2.5.4.jar
[info] Including from cache: scala-library-2.11.7.jar
[info] Including from cache: httpcore-4.3.2.jar
[info] Including from cache: postgresql-9.4-1201-jdbc41.jar
[info] Including from cache: commons-codec-1.6.jar
[info] Including from cache: slf4j-nop-1.6.4.jar
[info] Including from cache: amazon-kinesis-producer-0.10.1.jar
[info] Including from cache: joda-time-2.8.2.jar
[info] Including from cache: HikariCP-2.3.8.jar
[info] Including from cache: chill_2.11-0.7.0.jar
[info] Including from cache: chill-java-0.7.0.jar
[info] Including from cache: javassist-3.18.2-GA.jar
[info] Including from cache: kryo-2.21.jar
[info] Including from cache: guava-18.0.jar
[info] Including from cache: slick-pg_2.11-0.9.1.jar
[info] Including from cache: reflectasm-1.07-shaded.jar
[info] Including from cache: slf4j-api-1.7.12.jar
[info] Including from cache: asm-4.0.jar
[info] Including from cache: slick-pg_core_2.11-0.9.1.jar
[info] Including from cache: minlog-1.2.jar
[info] Including from cache: protobuf-java-2.6.1.jar
[info] Including from cache: slf4j-simple-1.7.12.jar
[info] Including from cache: objenesis-1.2.jar
[info] Including from cache: commons-io-2.4.jar
[info] Including from cache: flyway-core-3.2.1.jar
[info] Including from cache: commons-lang-2.6.jar
[info] Including from cache: aws-java-sdk-core-1.9.37.jar
[info] Including from cache: commons-logging-1.1.3.jar
[info] Including from cache: play-json_2.11-2.4.2.jar
[info] Including from cache: play-iteratees_2.11-2.4.2.jar
[info] Including from cache: httpclient-4.3.4.jar
[info] Including from cache: play-datacommons_2.11-2.4.2.jar
[info] Including from cache: joda-convert-1.7.jar
[info] Including from cache: scala-stm_2.11-0.7.jar
[info] Including from cache: scala-reflect-2.11.6.jar
[info] Including from cache: config-1.3.0.jar
[info] Including from cache: jackson-core-2.5.4.jar
[info] Including from cache: jackson-annotations-2.5.4.jar
[info] Including from cache: slick_2.11-3.0.0.jar
[info] Including from cache: play-functional_2.11-2.4.2.jar
[info] Including from cache: reactive-streams-1.0.0.jar
[info] Including from cache: jackson-databind-2.5.4.jar
[info] Including from cache: jackson-datatype-jdk8-2.5.4.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/DEPENDENCIES' with strategy 'discard'
[warn] Merging 'META-INF/INDEX.LIST' with strategy 'discard'
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/amazon-kinesis-producer/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/amazon-kinesis-producer/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/aws-java-sdk-core/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/aws-java-sdk-core/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.kryo/kryo/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.kryo/kryo/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.minlog/minlog/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.minlog/minlog/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.reflectasm/reflectasm/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.reflectasm/reflectasm/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jdk8/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jdk8/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.guava/guava/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.guava/guava/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.protobuf/protobuf-java/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.protobuf/protobuf-java/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.zaxxer/HikariCP/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.zaxxer/HikariCP/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-codec/commons-codec/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-codec/commons-codec/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-io/commons-io/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-io/commons-io/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-lang/commons-lang/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-lang/commons-lang/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-logging/commons-logging/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-logging/commons-logging/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/joda-time/joda-time/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/joda-time/joda-time/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpclient/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpclient/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpcore/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpcore/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.flywaydb/flyway-core/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.flywaydb/flyway-core/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.javassist/javassist/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.javassist/javassist/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-nop/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-nop/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-simple/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-simple/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/services/com.fasterxml.jackson.databind.Module' with strategy 'filterDistinctLines'
[error] 5 errors were encountered during merge
                                                                        [info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Assembly up to date: /home/user/Downloads/ScalaProj/datasembly/galactus/target/scala-2.11/galactus-assembly-0.1.0.jar
[trace] Stack trace suppressed: run last redshiftConnector/*:assembly for the full output.
[error] (redshiftConnector/*:assembly) deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /home/user/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /home/user/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-nop/jars/slf4j-nop-1.6.4.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-simple/jars/slf4j-simple-1.7.12.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-nop/jars/slf4j-nop-1.6.4.jar:org/slf4j/impl/StaticMDCBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-simple/jars/slf4j-simple-1.7.12.jar:org/slf4j/impl/StaticMDCBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-nop/jars/slf4j-nop-1.6.4.jar:org/slf4j/impl/StaticMarkerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-simple/jars/slf4j-simple-1.7.12.jar:org/slf4j/impl/StaticMarkerBinder.class
[error] Total time: 41 s, completed Oct 1, 2015 5:21:47 PM

here is my build.sbt files

import com.amazonaws.services.s3.model.Region
import ohnosequences.sbt.SbtS3Resolver.autoImport._

name := "h"

lazy val commonSettings = Seq(
  organization := "datasembly",
  version := "0.1.0-SNAPSHOT",
  scalaVersion := "2.11.7",
  resolvers += "akka" at "http://repo.akka.io/snapshots",
  resolvers += Resolver.sonatypeRepo("public"),
  publishMavenStyle := false,
  s3region := Region.US_Standard,
  awsProfile := "maven",
  publishTo := {
    val prefix = if (isSnapshot.value) "snapshots" else "releases"
    Some(s3resolver.value("My "+prefix+" S3 bucket", s3(prefix+".mvn-repo.d")) withIvyPatterns)
  }
)

lazy val commonDepends = Seq(
  libraryDependencies ++= Seq(
    "com.github.scopt"            %%  "scopt"                     %   "3.3.0"
  )
)

lazy val clusterDepends = Seq(
  libraryDependencies ++= Seq(
    "com.typesafe.akka"       %%  "akka-cluster"      %   "2.4-M2",
    "com.twitter"             %%  "chill-all"         %   "0.7.0",
    "com.twitter"             %%  "chill-akka"        %   "0.7.0"
  )
)

lazy val root = (project in file(".")) aggregate(core, worker, master, slick, aws) settings(commonSettings: _*)

lazy val core = project settings(commonSettings: _*) settings(clusterDepends: _*)

lazy val worker = project dependsOn core settings(commonSettings: _*) settings(commonDepends: _*) settings(clusterDepends: _*) enablePlugins JavaAppPackaging enablePlugins UniversalPlugin

lazy val master = project dependsOn core settings(commonSettings: _*) settings(commonDepends: _*) settings(clusterDepends: _*) enablePlugins JavaAppPackaging

lazy val client = project dependsOn core settings(commonSettings: _*) settings(commonDepends: _*) enablePlugins JavaAppPackaging

// Extensions

lazy val aws = project in file("extensions/aws") settings(commonSettings: _*) dependsOn core

// Contrib

lazy val slick = project in file("contrib/slick") settings(commonSettings: _*) dependsOn core

bulid.sbt

name := "h-core"

libraryDependencies ++= Seq(
  "org.apache.commons"          %   "commons-compress"          %   "1.9",
  "com.jsuereth"                %%  "scala-arm"                 %   "1.4"
)

build.sbt

name := "h-client"

libraryDependencies ++= Seq(
  "jline"                       %   "jline"           %   "2.12",
  "com.typesafe.play"           %%  "play-ws"         %   "2.4.1"
)

build.sbt name := "h-master"

libraryDependencies ++= Seq(
  "com.typesafe.play"           %%    "play"                      %   "2.4.1",
  "com.typesafe.play"           %%    "play-netty-server"         %   "2.4.1",
  "com.leansoft"                %     "bigqueue"                  %   "0.7.0"
)

resolvers += "github.release.repo" at "https://raw.githubusercontent.com/bulldog2011/bulldog-repo/master/repo/releases/"

checksums in update := Nil

Update 2 tried the solution given my user rosshsr but error remain same

libraryDependencies +=  excludeAll(

  ExclusionRule("com.esotericsoftware.minlog" , "minlog"),
  ExclusionRule("org.slf4j" ,   "slf4j-nop"),
  ExclusionRule("com.esotericsoftware.kryo" % "kryo")

)

Please help

Oh jeez I've dealt with this before. Can you post your build.sbt please? — turbo_laser, Oct 01 '15 at 13:19

score 1 · Answer 1 · answered Oct 01 '15 at 13:46

Since I'm not familiar with where in all of these libraries you're depending on these sets of clashes are occurring, I'll just give you my general strategy for fixing this.

You need to exclude one of those clashing dependencies.

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.0.0-alpha" excludeAll(

  ExclusionRule("commons-logging","commons-logging-api"),
  ExclusionRule("asm","asm")

)

This is an excerpt from one of my build.sbt files.

You would need to attach that excludeAll bit to the dependency and add something like ExclusionRule("org.slf4j","slf4j-simple")

please sea my updated i tried but error remains same – swaheed Oct 01 '15 at 14:17 — swaheed, Oct 01 '15 at 14:17

sbt assembly command shows error: (redshiftConnector/*:assembly) deduplicate: different file contents found

1 Answers1