0

The Spark version is 1.3.0.

The source code from the SQLContext.scala (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala):

 @transient
  protected[sql] val ddlParser = new DDLParser(sqlParser.apply(_))

  @transient
  protected[sql] val sqlParser = {
    val fallback = new catalyst.SqlParser
    new SparkSQLParser(fallback(_))
  }

  protected[sql] def parseSql(sql: String): LogicalPlan = {
    ddlParser(sql, false).getOrElse(sqlParser(sql))
  }

I cannot really understand the code above.

  • how does the sqlParser.apply(_) work ?

what is (_) on earth? DDLParser's constructor needs a parameter parseQuery: String => LogicalPlan, but the return type of sqlParser.apply is LogicalPlan.

  • similarly, fallback(_) is also a apply call, how does (_) work?
chenzhongpu
  • 6,193
  • 8
  • 41
  • 79

1 Answers1

1

The underscore in this case denotes a partial function application, which is one way to convert a method type into a function type. So when you write sqlParser.apply(_), it is equivalent to x => sqlParser.apply(x). Since the DDLParser constructor needs an argument of type String => LogicalPlan, and sqlParser.apply returns a LogicalPlan, the compiler can deduce that x here is actually a String, and so sqlParser.apply(_) is of type String => LogicalPlan.

You have many options here. The following would be equivalent under most circumstances:

new DDLParser(x => sqlParser.apply(x))
new DDLParser(x => sqlParser(x))
new DDLParser(sqlParser.apply(_))
new DDLParser(sqlParser.apply _)
new DDLParser(sqlParser(_))
new DDLParser(sqlParser _)
new DDLParser(sqlParser.apply)
new DDLParser(sqlParser)

You can read more about the difference between methods types and function types here.

Another, closely related way of turning a method type into a function type is called eta expansion, which is used interchangeably in certain contexts. Read about partial function application, eta expansion, and all the other uses of underscore in Scala here.

Community
  • 1
  • 1
Ben Reich
  • 16,222
  • 2
  • 38
  • 59
  • what confuses me is that : `val ddlParser = new DDLParser(sqlParser.apply(_))`, when I define such a `ddlParser`, I didn't pass any `x`(String type) to it. How does it do initialization work? – chenzhongpu Apr 15 '15 at 12:36
  • The constructor of `DDLParser` needs an argument of type `String => LogicalPlan`. The constructor doesn't need a `String`, but a function that that in a `String` and returns a `LogicalPlan`. And `sqlParser.apply(_)` returns exactly that type. – Ben Reich Apr 15 '15 at 13:26