0

I am quite new to Scala (and Spark, if this is somehow Spark-specific), so please forgive the super simple question.

To me, it seems like this code should compile just fine:

sqlContext.udf.register("json_extract_string", (rawJson: String, keyPath: String*) => {
    [String]UDFs.jsonExtract(rawJson, keyPath:_*)
})

Yet compiling gives the error:

Error:(31, 89) ')' expected but identifier found.
        sqlContext.udf.register("json_extract_string", (rawJson: String, keyPath: String*) => {
                                                                                        ^

Why is this?

The function being called looks like this:

object UDFs {
    def jsonExtract[T: Manifest](rawJson: String, keyPath: String*): Option[T] = {
        implicit val formats = DefaultFormats
        val json = parse(rawJson)
        keyPath.foldLeft(json)(_ \ _).extractOpt[T]
    }
}
obeattie
  • 3,264
  • 2
  • 31
  • 36

2 Answers2

1

This:

[String]UDFs.jsonExtract(rawJson, keyPath:_*)

is not valid Scala.

If you need to cast, you have to explicitly call asInstanceOf:

UDFs.jsonExtract(rawJson, keyPath:_*).asInstanceOf[String]

But typically such casting is a code smell and a sign that you've gone down the wrong path.

Ryan
  • 7,227
  • 5
  • 29
  • 40
  • I think given `jsonExtract`, OP will want to do `UDFs.jsonExtract[String](rawJson, keyPath:_*)` – Daenyth Jan 11 '16 at 20:58
  • This seems to give the exact same error. Here is a minimal test case that causes the compilation error. I feel I must be missing something ridiculously simple. https://gist.github.com/obeattie/d8a0eb9ed860b5023f63 – obeattie Jan 11 '16 at 21:17
1

In scala it is not permitted for anonymous functions to have variable length arguments, see this answer Scala: How do I define an anonymous function with a variable argument list?

There is a shorter form of what you're trying to express which should work:

sqlContext.udf.register("json_extract_string", UDFs.jsonExtract[String]_)
Community
  • 1
  • 1
mattinbits
  • 10,370
  • 1
  • 26
  • 35