2

Currently I face some problems with shapeless in spark 2.1 where still an outdated version of shapeless is being used. One of my dependencies(PureConfig current version of 0.8.0) pulls in a newer version of shapeless.

One should be able to use:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll
)

to fix the method not found problem - but that does not help.

Also

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.8.0")
    .inProject
)

from Spark not working with pureconfig is suggested (and in my opinion should not be better than the inAll option) but also fails (with a different error as this is now breaking the library (pureconfig).

Any suggestions to fix the shapeless problem?

edit

currently, shading configuration looks like:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" % "pureconfig_2.11" % pureconfigVersion)
    .inLibrary("com.github.pureconfig" % "pureconfig-macros_2.11" % pureconfigVersion)
    .inProject
)
Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
  • ‘InAll’ means you’ll be scanning your entire dependency graph which can be time consuming as the graph gets large. You must likely don’t want that. – Yuval Itzchakov Dec 09 '17 at 17:08
  • What happens when you try specifing the Scala version. i.e. .inLibrary("com.github.pureconfig" % "pureconfig_2.11" % "0.8.0") – Yuval Itzchakov Dec 09 '17 at 17:12
  • That indeed has some effect - but for now only leads to a lot of merge errors: `[error] /Users/geoheil/.ivy2/cache/com.github.pureconfig/pureconfig_2.11/bundles/pureconfig_2.11-0.8.0.jar:pureconfig/Derivation$.class [error] /Users/geoheil/.ivy2/cache/com.github.pureconfig/pureconfig-macros_2.11/jars/pureconfig-macros_2.11-0.8.0.jar:pureconfig/Derivation$.class [error] deduplicate: different file contents found in the following:` but this should not have any different effect than `inAll` – Georg Heiler Dec 09 '17 at 17:21
  • Adding ` .inLibrary("com.github.pureconfig" % "pureconfig-macros_2.11" % pureconfigVersion) ` will fix it locally. Only on Monday I will be able to confirm whether it works in yarn cluster mode - assuming the current shade settings produce the same output as `inAll` it probably won't work. I will also try to experiment with `userclasspathFirst=true`. – Georg Heiler Dec 09 '17 at 17:29
  • Don't go with userClassPathFirst, that only leads to trouble. – Yuval Itzchakov Dec 09 '17 at 19:13
  • So you meant best would be to switch to a non shapeless based library for now? – Georg Heiler Dec 09 '17 at 19:48
  • No, we just need to figure this out :) – Yuval Itzchakov Dec 09 '17 at 19:56
  • But what else to shade ;) in case you have a YARN cluster at hand - maybe you can try: https://github.com/geoHeil/sparkMiniKickstart – Georg Heiler Dec 09 '17 at 19:59
  • Problem is im not near a computer now. Update on Monday if this works. If not, will try to dive into it deeper. – Yuval Itzchakov Dec 09 '17 at 20:34
  • 1
    Thanks. Meanwhile, I set up a small projec to reproduce my problem: git clone git@github.com:geoHeil/sparkMiniKickstart.git cd sparkMiniKickstart sbt compile – Georg Heiler Dec 09 '17 at 20:48

1 Answers1

1

Not really an answer but from spark 2.2 onwards it works fine as sparks dependency was upgraded.

Georg Heiler
  • 16,916
  • 36
  • 162
  • 292