Spark not working with pureconfig

Question

I'm trying to use pureConfig and configFactory for my spark application configuration. here is my code:

import pureconfig.{loadConfigOrThrow}
object Source{
  def apply(keyName: String, configArguments: Config): Source = {
    keyName.toLowerCase match {
      case "mysql" =>
          val properties = loadConfigOrThrow[DBConnectionProperties](configArguments)
          new MysqlSource(None, properties)
      case "files" =>
        val properties = loadConfigOrThrow[FilesSourceProperties](configArguments)
        new Files(properties)
      case _ => throw new NoSuchElementException(s"Unknown Source ${keyName.toLowerCase}")
    }

  }
}

import Source
val config = ConfigFactory.parseString(result.mkString("\n"))
    val source = Source("mysql",config.getConfig("source.mysql"))

when I run it from the IDE (intelliJ) or directly from java (i.e java jar...) it works fine.

But when I run it with spark-submit it fails with the following error:

Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;

From a quick search I found a similar similar to this question. which suggest the reason for this is due to the fact both spark and pureConfig depends on Shapeless but with different versions,

I tried to shade it as suggested in the answer

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0").inProject
)

but it didn't work as well can it be from a different reason? any idea what may work?

Thanks

Yuval Itzchakov · Accepted Answer · 2017-04-26T05:18:21.103

5

You also have to shade shapeless inside its own JAR, in addition to pureconfig:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0")
    .inProject
)

Make sure to add the correct shapeless version.

edited Apr 26 '17 at 05:18

answered Apr 25 '17 at 18:02

Yuval Itzchakov

146,575
32
257
321

1

Coz of a [bug in sbt-assembly](https://github.com/sbt/sbt-assembly/issues/235) you have to add `scalaVersion` suffix to the library matching code mentioned in `inLibrary` like ` .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")` – kaychaks Apr 26 '17 at 04:50
@kaychaks I actually ran into that and managed to forget. Thanks for the tip, updated the answer. – Yuval Itzchakov Apr 26 '17 at 05:55
@YuvalItzchakov I think spark uses shapeless 2.0.0 via breeze (http://apache-spark-developers-list.1001551.n3.nabble.com/shapeless-in-spark-2-1-0-td20392.htm) when I ran your fix I got the following error `java.lang.NoClassDefFoundError: shapeless/DefaultSymbolicLabelling` should I shade spark shapeless dependency instead? – Lior Baber Apr 26 '17 at 07:02
@LiorBaber No, you need to shade your own so your project picks up the correct version *you* need to work with. – Yuval Itzchakov Apr 26 '17 at 07:05
@YuvalItzchakov I did, but it failed with the error `java.lang.NoClassDefFoundError: shapeless/DefaultSymbolicLabelling` here is what I did: `libraryDependencies += "com.github.pureconfig" %% "pureconfig" % "0.7.0" libraryDependencies += "com.chuusai" % "shapeless_2.11" % "2.3.2" assemblyShadeRules in assembly := Seq( ShadeRule.rename("shapeless.**" -> "shadeshapless.@1") .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2") .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0") .inProject )` – Lior Baber Apr 26 '17 at 07:26
Did you open the JAR? Was the library properly shaded? – Yuval Itzchakov Apr 26 '17 at 07:28
how can I know that? should I extract the jar and look in the pureconfig classes? – Lior Baber Apr 26 '17 at 07:30
Moreover, the issue happened only when I use spark-submit meaning the spark dependencies are not included in the assembly jar (in Porvided scope) – Lior Baber Apr 26 '17 at 07:36
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/142701/discussion-between-yuval-itzchakov-and-lior-baber). – Yuval Itzchakov Apr 26 '17 at 08:45
Is there an answer to this question? I am facing the same problem for `0.8.0` https://github.com/pureconfig/pureconfig/issues/337 and cant get it to work in any of the suggested solutions. – Georg Heiler Dec 08 '17 at 13:37
@GeorgHeiler You said in the issue that YARN client mode works fine. Did you properly distribute the shaded jars? – Yuval Itzchakov Dec 08 '17 at 22:38
They are already included in the jar with the code for the job. – Georg Heiler Dec 09 '17 at 06:43
Also getting `java.lang.NoClassDefFoundError: shapeless/DefaultSymbolicLabelling` when using the shading recommendation for maven. – horatio1701d Dec 09 '17 at 11:50

Spark not working with pureconfig

1 Answers1

Linked