Why does the Spark application code need to be an object rather than a class?

Question

I ran into Exception in thread "Driver" java.lang.NullPointerException while running my Spark application, and it was because my code was inside a class rather than an object, as suggested here. However, just wondering why the main class has to be an object. Is it possible to use a class instead?

Answer 1

This is an issue with Scala itself, not with Spark. When a Scala class is compiled the methods inside are not converted into the equivalent of Java static methods. This means that the main method will not be static and the JVM will hence not be able to find it when executing the program. In contrast, all methods inside a Scala object will be static. A good explaination of the differences can be found here.

It is not possible to only use a single class since the main method needs to be in an object. However, it would be possible to have an object that simply creates an instance of the class and all other code in the class - but it seems unnecessary when you can simply make the class into an object instead.

Answer 2

I'll answer your question in sequence:

Why the main class has to be an object

This is because static members (methods or fields) do not exist in Scala. Rather than defining static members, the Scala declares these members in singleton objects.

An object is a class that has exactly one instance, which is created lazily when it is referenced. Hence, when we declare Object we declare a singleton object.

Is it possible to use a class instead

No.

In addition to help understand the concept, you have to understand the concept of companion object, which is an object with the same name as a class.

Using this we can declare the object with same name as class and then declare the static members there.

For e.g. in below code I'm keeping the header static.

class HelloWorld(msg:String) {
    import HelloWorld._  //import static members

    def msg():String = {
        getHeader + msg //append static header
    }
}

object HelloWorld {  //companion object
  private def getHeader:String = "Hello, "  //static method

  def main(args: Array[String]) {
    val scalaObj = new HelloWorld("scala!")
    val pyObj = new HelloWorld("Python!") 

    println(scalaObj.msg)
    println(pyObj.msg)

  }
}

Output:

Hello, scala!
Hello, python!