0

I am hitting this error and unable to find any resolution so far. using Intellj Intellj community version 2019.2.3 and maven for bulid

Scenario :

  1. Module A has A,B,C css class definition files defining the schema for the datasets
  2. Module B has dependency on Module A with dependency defined in the POM
  3. Module B is the main Module . I am getting a this error

User class threw exception: scala.ScalaReflectionException: class "C" in JavaMirror with org.apache.spark.util.MutableURLClassLoader@40021799 of type class org.apache.spark.util.MutableURLClassLoader with classpath file:/mnt/resource/hadoop/yarn/local/usercache/livy/appcache/application_1610888522323_13133/container_e03_1610888522323_13133_01_000001/app.jar] and parent being sun.misc.Launcher$AppClassLoader@306a30c7 of type class sun.misc.Launcher$AppClassLoader with classpath

Code is val temp:

Dataset[A] = getFoo() 
Val slimDataset: Dataset[C] = temp.Select($"ColA",$"colN").as[C]

This line is throwing an exception saying unable to load the file. I opened Module A.jar which has Classes A and B from Module B but C is missing.

Any thoughts on why Intellj is not including all the libraries (in this case class C from Module B)?

blackbishop
  • 30,945
  • 11
  • 55
  • 76
novice8989
  • 149
  • 1
  • 11
  • I figured minimize jar option is excluding some of the class files from the Module B above. true . It seems we can mention filters in this config section to tell to maven not to exclude the class files. . Is it the artifiact Id ? Any pointers on how to use thee filers to include module please ? – novice8989 Feb 10 '21 at 10:37
  • I came across this https://stackoverflow.com/questions/8817257/minimize-an-uber-jar-correctly-using-shade-plugin , but i am getting security exception in the manifest when i tired to follow the suggestion mentioned in the above thread. I am not able to ask on that thread as I dont have enough reputation points :( . Any help on this will be appreciated – novice8989 Feb 10 '21 at 21:33

0 Answers0