Scala Reflection Instantiating class issues with embded external library in this spark schema structs

Question

I saw lots of websites about scala reflection library but none of them have a straightforward answer to instantiate an object of the class at runtime. For example, I have the following code:

trait HydraTargetTable {
 val inputTables = Seq.empty[Int]
 val tableType: String
 val tableName: String
 val schema: StructType
 def className: String = this.getClass.toString
}

trait HydraIntermediateTable extends HydraTargetTable {
  val tableType = "Intermediate"

  def build(textRDD: RDD[String]): DataFrame = {
    DataframeUtils.safeParseFromSchema(textRDD, schema)
  }
}
class Table1 extends HydraIntermediateTable {
  override val inputTables: Seq[Int] = Seq(1, 2)
  override val tableName: String = ""
 override val schema: StructType = new StructType()
}

At runtime, I want to be able to instantiate an object of Table1 given the class name as a String value. Here is my reflection code.

object ReflectionTestApp {

  def tableBuilder(name: String): Intermediate = {
    Class.forName("hearsay.hydra.dataflow.api." + name).newInstance()
     .asInstanceOf[Intermediate]
  }

  def hydraTableBuilder(name: String): HydraTargetTable = {
    val action = Class
     .forName("hearsay.hydra.dataflow.api." + name).newInstance()
    action.asInstanceOf[HydraTargetTable]
  }

  def main(args: Array[String]): Unit = {
   hydraTableBuilder("Table1").inputTables.foreach(println)
  }

 }

Possible duplicate of [Scala instantiate objects from String classname](https://stackoverflow.com/questions/32384370/scala-instantiate-objects-from-string-classname) — vindev, Mar 16 '18 at 05:15

vindev · Accepted Answer · 2018-03-16T06:30:58.403

0

Here is how you can achieve reflection for Object/Class.

package reflection

trait Table {
  val id: Int
}

class ActivityTable extends Table {
  val id = 10
}

object ActivityTable2 extends Table {
  val id = 10
}

object Reflection extends App {

  val obj = activityTableBuilder("ActivityTable")
  println(obj.id) //output 10

  val obj2 = objectBuilder("ActivityTable2$")
  println(obj2.id) //output 10

  /*
  class reflection
   */
  def activityTableBuilder(name: String): Table = {
    val action = Class.forName("reflection." + name).newInstance()
    action.asInstanceOf[Table]
  }

  /*
  object reflection
   */
  def objectBuilder(name: String): Table = {
    val action = Class.forName("reflection." + name)
    action.getField("MODULE$").get(classOf[Table]).asInstanceOf[Table]
  }
}

edited Mar 16 '18 at 06:30

answered Mar 16 '18 at 06:22

vindev

2,240
2
13
20

Thank you. I came across a new problem where the external library is being used and reflection library is not able find the class. – Shivakanth Komatreddy Mar 16 '18 at 06:38
I ran your code in my machine(commented tableBuilder and build method) and it works fine. I think its some dependency issue as you are getting java.lang.ClassNotFoundException. – vindev Mar 16 '18 at 06:52
I know it is a dependency issue. It's the line where I am initializing the schema val with new Struct(). I added the exception that I am getting and according to that, it's coming from the apache spark sql library which is an external library that I am using. So I don't know how to load thoses classes. – Shivakanth Komatreddy Mar 16 '18 at 07:23
I just got it working, I ran it on spark shell and it worked, I guess I messed up by not testing the script in the spark shell. – Shivakanth Komatreddy Mar 16 '18 at 07:54

Scala Reflection Instantiating class issues with embded external library in this spark schema structs

1 Answers1