0

I have a spark project using Janusgraph that I am trying to making a fat jar of. The projects runs well in IntelliJ but when I try to make a fat jar and build it and run it with spark-submit, it throws this error:

NoSuchMethodError:com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch

I am creating a fat jar using sbt clean compile and sbt assembly.

I also unzipped the jar created and looked for the Stopwatch.class and the method mentioned. They seem to be present there. So, I don't understand why this error would come up.

My build.sbt file is as follows:

name := "proj1"

version := "0.1"

scalaVersion := "2.11.12"


// https://mvnrepository.com/artifact/com.michaelpollmeier/gremlin-scala
libraryDependencies += "com.michaelpollmeier" %% "gremlin-scala" % "3.3.1.1"
// https://mvnrepository.com/artifact/org.janusgraph/janusgraph-core
libraryDependencies += "org.janusgraph" % "janusgraph-core" % "0.2.0"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-mllib
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.3.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-hive
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.3.0"

//libraryDependencies += "com.google.guava" %% "guava" % "15.0"
//libraryDependencies += "org.apache.hadoop" %% "hadoop-client" % "2.7.2"
// https://mvnrepository.com/artifact/org.janusgraph/janusgraph-es
libraryDependencies += "org.janusgraph" % "janusgraph-es" % "0.2.0"

// https://mvnrepository.com/artifact/org.janusgraph/janusgraph-cassandra
libraryDependencies += "org.janusgraph" % "janusgraph-cassandra" % "0.2.0"

// https://mvnrepository.com/artifact/com.google.guava/guava
libraryDependencies += "com.google.guava" % "guava" % "24.0-jre"

// https://mvnrepository.com/artifact/commons-io/commons-io
libraryDependencies += "commons-io" % "commons-io" % "2.6"



assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

Any help is appreciated. Thanks in advance!

J.Doe
  • 183
  • 1
  • 13
  • Possible duplicate of [NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch](https://stackoverflow.com/questions/28855559/nosuchmethoderror-com-google-common-base-stopwatch-createstartedlcom-google-c) – Ramesh Maharjan Mar 09 '18 at 07:10
  • The solution there was to update the guava version. I am already using the latest one. – J.Doe Mar 09 '18 at 07:11
  • According to the [version compatibility matrix](http://docs.janusgraph.org/latest/version-compat.html), JanusGraph 0.2.0 is compatible with Spark 1.6.z. Also note JanusGraph is compatible with Scala 2.10.z. Try aligning versions with JanusGraph instead of taking the latest available. – Jason Plurad Mar 11 '18 at 22:07
  • The projects run well on Intellij. Only when I try to build a fat jar, it shows this error. Is this because some library is importing an old version of guava and thus it cannot find that particular method? I tried printing the dependency tree, at all places where there is a lower version of guava being used, it's written `(evicted by: 24.0-jre)`. So, then, why would something like this happen? – J.Doe Mar 12 '18 at 09:06

0 Answers0