Why spark-shell throws NoSuchMethodException while calling newInstance via reflection

Question

spark-shell throws NoSuchMethodException if I define a class in REPL and then call newInstance via reflection.

Spark context available as 'sc' (master = yarn, app id = application_1656488084960_0162).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.3
      /_/
         
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.
Type :help for more information.

scala> class Demo {
     |   def demo(s: String): Unit = println(s)
     | }
defined class Demo

scala> classOf[Demo].newInstance().demo("OK")
java.lang.InstantiationException: Demo
  at java.lang.Class.newInstance(Class.java:427)
  ... 47 elided
Caused by: java.lang.NoSuchMethodException: Demo.<init>()
  at java.lang.Class.getConstructor0(Class.java:3082)
  at java.lang.Class.newInstance(Class.java:412)
  ... 47 more

But the same code works fine in native scala REPL:

Welcome to Scala 2.12.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> class Demo {
     |   def demo(s: String): Unit = println(s)
     | }
defined class Demo

scala> classOf[Demo].newInstance().demo("OK")
OK

What's the difference between spark-shell REPL and native scala REPL?

I guess the Demo class might be treated as inner class in spark-shell REPL.

But ... how to solve the problem?

Answer 1

In Scala 2.12.4 REPL the class is nested into objects, so it has zero-argument constructor accessible via .newInstance(). In Spark 3.0.3 shell the class is nested into classes, so there is no zero-argument constructor, the constructor of Demo accepts an instance of outer class and should be accessed via .getConstructors()(0).newInstance(...). Start Scala REPL and Spark shell with ./scala -Xprint:typer and ./spark-shell -Xprint:typer correspondingly and you'll see the difference.

So in Spark shell try

classOf[Demo].getDeclaredMethod("demo", classOf[String])
  .invoke(
    classOf[Demo].getConstructors()(0).newInstance($lineXX.$read.INSTANCE.$iw.$iw), 
    "OK"
  )

//OK
//resYY: Object = null

(XX is the number of line where Demo is defined).

See details in Spark adds hidden parameter to constructor of a Scala class