Assembly scala project causes deduplicate errors

Question

I'm trying to assembly my scala project and cant get rid of some deduplicate errors Here is the problematic output:

> [error] 2 errors were encountered during merge [error] stack trace is
> suppressed; run 'last
> ProjectRef(uri("https://hyehezkel@fs-bitbucket.fsd.forescout.com/scm/~hyehezkel/classification_common.git#test_branch"),
> "global") / assembly' for the full output [error]
> (ProjectRef(uri("https://hyehezkel@fs-bitbucket.fsd.forescout.com/scm/~hyehezkel/classification_common.git#test_branch"),
> "global") / assembly) deduplicate: different file contents found in
> the following: [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-buffer\4.1.42.Final\netty-buffer-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-codec\4.1.42.Final\netty-codec-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-common\4.1.42.Final\netty-common-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-handler\4.1.42.Final\netty-handler-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-resolver\4.1.42.Final\netty-resolver-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-transport-native-epoll\4.1.42.Final\netty-transport-native-epoll-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-transport-native-unix-common\4.1.42.Final\netty-transport-native-unix-common-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\io\netty\netty-transport\4.1.42.Final\netty-transport-4.1.42.Final.jar:META-INF/io.netty.versions.properties
> [error] deduplicate: different file contents found in the following:
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\fasterxml\jackson\core\jackson-annotations\2.10.1\jackson-annotations-2.10.1.jar:module-info.class
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\fasterxml\jackson\core\jackson-core\2.10.1\jackson-core-2.10.1.jar:module-info.class
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\fasterxml\jackson\core\jackson-databind\2.10.1\jackson-databind-2.10.1.jar:module-info.class
> [error]
> C:\Users\hyehezkel\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\fasterxml\jackson\dataformat\jackson-dataformat-csv\2.10.0\jackson-dataformat-csv-2.10.0.jar:module-info.class

I have read the following article but didnt manage to solve it: https://index.scala-lang.org/sbt/sbt-assembly/sbt-assembly/0.14.5?target=_2.12_1.0

This is my plugins.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")

And this is my build.st

import sbt.Keys.{dependencyOverrides, libraryDependencies, mappings}
import sbtassembly.AssemblyPlugin.assemblySettings._

name := "classification_endpoint_discovery"
version := "0.1"
organization in ThisBuild := "com.forescout"
scalaVersion in ThisBuild := "2.13.1"
updateOptions := updateOptions.value.withCachedResolution(true)
//classpathTypes += "maven-plugin"
exportJars := true
logLevel := Level.Info
logLevel in assembly := Level.Debug
lazy val commonProject = RootProject(uri("https://hyehezkel@fs-bitbucket.fsd.forescout.com/scm/~hyehezkel/classification_common.git#test_branch"))

lazy val global = project
  .in(file("."))
  .settings(settings)
  .enablePlugins(AssemblyPlugin)
//  .disablePlugins(AssemblyPlugin)
  .aggregate(
    commonProject,
    `endpoint-discovery`
  )

lazy val `endpoint-discovery` = project
  .settings(
    name := "endpoint-discovery",
    settings,
    assemblySettings,
    assemblyJarName in assembly := "endpoint-discovery.jar",
    assemblyJarName in assemblyPackageDependency := "endpoint-discovery-dep.jar",
    libraryDependencies += dependencies.postgresql,
    libraryDependencies += "com.lihaoyi" %% "ujson" % "0.7.5",
    libraryDependencies += "com.lihaoyi" %% "requests" % "0.2.0",
    libraryDependencies += dependencies.`deepLearning4j-core`,
    libraryDependencies += dependencies.`deeplearning4j-nn`,
    libraryDependencies += dependencies.`nd4j-native-platform`,
    excludeDependencies += "commons-logging" % "commons-logging"
//    dependencyOverrides += "org.slf4j" % "slf4j-api" % "1.7.5",
//    dependencyOverrides += "org.slf4j" % "slf4j-simple" % "1.7.5",
  )
  .dependsOn(commonProject)
  .enablePlugins(AssemblyPlugin)
lazy val dependencies =
  new {
    val deepLearning4jV         = "1.0.0-beta4"
    val postgresqlV             = "9.1-901.jdbc4"

    val `deepLearning4j-core`    = "org.deeplearning4j"         % "deeplearning4j-core"      % deepLearning4jV
    val `deeplearning4j-nn`      = "org.deeplearning4j"         % "deeplearning4j-nn"        % deepLearning4jV
    val `nd4j-native-platform`   = "org.nd4j"                   % "nd4j-native-platform"     % deepLearning4jV
    val postgresql               = "postgresql"                 % "postgresql"               % postgresqlV
  }

// SETTINGS
lazy val settings =
  commonSettings

lazy val compilerOptions = Seq(
  "-unchecked",
  "-feature",
  "-language:existentials",
  "-language:higherKinds",
  "-language:implicitConversions",
  "-language:postfixOps",
  "-deprecation",
  "-encoding",
  "utf8"
)

lazy val commonSettings = Seq(
  scalacOptions ++= compilerOptions
)

lazy val assemblySettings = Seq(
  assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false),
  assemblyMergeStrategy in assembly := {
    case PathList("META-INF", "io.netty.versions.properties", xs @ _*) => MergeStrategy.singleOrError
    case "module-info.class"                         => MergeStrategy.singleOrError

    case PathList("org", "xmlpull", xs @ _*)         => MergeStrategy.discard
    case PathList("org", "nd4j", xs @ _*)            => MergeStrategy.first
    case PathList("org", "bytedeco", xs @ _*)        => MergeStrategy.first
    case PathList("org.bytedeco", xs @ _*)           => MergeStrategy.first
    case PathList("META-INF", xs @ _*)               => MergeStrategy.discard
    case "XmlPullParser.class"                       => MergeStrategy.discard
    case "Nd4jBase64.class"                          => MergeStrategy.discard
    case "XmlPullParserException.class"              => MergeStrategy.discard
//    case n if n.startsWith("rootdoc.txt") => MergeStrategy.discard
//    case n if n.startsWith("readme.html") => MergeStrategy.discard
//    case n if n.startsWith("readme.txt") => MergeStrategy.discard
    case n if n.startsWith("library.properties") => MergeStrategy.discard
    case n if n.startsWith("license.html") => MergeStrategy.discard
    case n if n.startsWith("about.html") => MergeStrategy.discard
//    case _ => MergeStrategy.first
      case x =>
        val oldStrategy = (assemblyMergeStrategy in assembly).value
        oldStrategy(x)
  }
)

I have tried many Merge Strategies but nothing works What am i missing here? Any advice?

score 1 · Answer 1 · answered May 25 '20 at 13:21

for META-INF/io.netty.versions.properties

you have:
```
case PathList("META-INF", "io.netty.versions.properties", xs @ _*) => MergeStrategy.singleOrError
```
which says that it will error out, if there are more than 1 files with this name.

try MergeStrategy.first for these files instead
module-info.class

these files are only relevant for the Java 9 module system. Usually, you can just discard them:
```
case "module-info.class"                       => MergeStrategy.discard
```

@so what does it mean? make a concrete example, with clear instructions how to reproduce. — cbley, May 26 '20 at 11:11

Assembly scala project causes deduplicate errors

1 Answers1