4

Why import implicit SqlContext.implicits._ after initializing SQLContext in a scala spark application.

when import is put outside the object , there seems to be an issue.

I am from a java back ground , where am not understanding the usage of import statements within a def object.

val sqlContext = new SQLContext(sc)

import sqlContext.implicits._
import sqlContext._
ashwinsakthi
  • 1,856
  • 4
  • 27
  • 56

1 Answers1

5

The import doesn't work externally because the implicits object is defined inside the SQLContext class:

* :: Experimental ::
* (Scala-specific) Implicit methods available in Scala for converting
* common Scala objects into `DataFrame`s.
*
* {{{
*   val sqlContext = new SQLContext(sc)
*   import sqlContext.implicits._
* }}}
@Experimental
@InterfaceStability.Evolving
object implicits extends SQLImplicits with Serializable {
  protected override def _sqlContext: SQLContext = self
}

Thus, only once you have an instance of SQLContext can you import the internal object definition containing all the implicits.

Yuval Itzchakov
  • 146,575
  • 32
  • 257
  • 321
  • 1
    So when i do an import org.apache.spark.sql.SQLContext, and create an instance of it. Will it not include all of the methods and objects defined inside ? Any objects defined inside a class will need to be explicitly imported in Scala. Only methods will be a part of the class instance and not objects defined within a class. That is my understanding as of now. Please correct me if i am wrong. – ashwinsakthi Feb 28 '17 at 09:12
  • 1
    @ashwinsakthi That is correct. It is not common for developers to put objects inside class definition (I haven't seen at least). Implicits are usually placed either inside companion objects or inside package objects. – Yuval Itzchakov Feb 28 '17 at 10:20