0

I am trying to compile and package a fat jar using SBT and I keep running into the following error. I have tried everything from using library dependency exclude and merging.

[trace] Stack trace suppressed: run last *:assembly for the full output.
[error] (*:assembly) deduplicate: different file contents found in the     following:
[error] /Users/me/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1    .7.10.jar:META-INF/maven/org.slf4j/slf4j-api/pom.properties
[error] /Users/me/.ivy2/cache/com.twitter/parquet-format/jars/parquet-format-2.2.0-rc1.jar:META-INF/maven/org.slf4j/slf4j-api/pom.properties
[error] Total time: 113 s, completed Jul 10, 2015 1:57:21 AM

The current incarnation of my build.sbt file is below:

import AssemblyKeys._

assemblySettings

name := "ldaApp"

version := "0.1"

scalaVersion := "2.10.4"

mainClass := Some("myApp")

libraryDependencies +="org.scalanlp" %% "breeze" % "0.11.2"

libraryDependencies +="org.scalanlp" %% "breeze-natives" % "0.11.2"

libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.3.1"

libraryDependencies +="org.ini4j" % "ini4j" % "0.5.4"

jarName in assembly := "myApp"

net.virtualvoid.sbt.graph.Plugin.graphSettings

libraryDependencies += "org.slf4j" %% "slf4j-api"" % "1.7.10" % "provided"

I realize I am doing something wrong...I just have no idea what.

Cœur
  • 37,241
  • 25
  • 195
  • 267
Ego
  • 585
  • 1
  • 8
  • 18
  • 2
    what is so maddenning? You are trying to create a fat jar and there're duplicates in the classpath. I think for the sbt assembly there's an option to ignore duplicates – vitalii Jul 10 '15 at 07:54
  • see here http://stackoverflow.com/questions/25144484/sbt-assembly-deduplication-found-error – vitalii Jul 10 '15 at 07:55
  • @user405887: why not include (in your question) what you already tried? That way people that try to think along won't present you that option. I see no sleights here, please assume good faith here. – Martijn Pieters Jul 11 '15 at 21:45

2 Answers2

0

Here is how you can handle these merge issues.

import sbtassembly.Plugin._

lazy val assemblySettings = sbtassembly.Plugin.assemblySettings ++ Seq(
    publishArtifact in packageScala := false, // Remove scala from the uber jar
    mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
        {
            case PathList("META-INF", "CHANGES.txt") => MergeStrategy.first
            // ...
            case PathList(ps @ _*) if ps.last endsWith "pom.properties" => MergeStrategy.first
            case x => old(x)
        }
    }
)

Then add these settings to your project.

lazy val projectToJar = Project(id = "MyApp", base = file(".")).settings(assemblySettings: _*)
marios
  • 8,874
  • 3
  • 38
  • 62
0

I got your assembly build running by removing spark from the fat jar (mllib is already included in spark).

libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.3.1" % "provided"

Like vitalii said in a comment, this solution was already here. I understand that spending hours on a problem without finding the fix can be frustrating but please be nice.

Community
  • 1
  • 1
David Duponchel
  • 3,959
  • 3
  • 28
  • 36