0

First we create a dataframe:

val df = sc.parallelize(Range(1, 100).map(x => (x * 1.0, x * 3.0, x * 0.7))).toDF("x1", "x2", "x3")

    df.show(3)

    //+---+---+------------------+
    //| x1| x2|                x3|
    //+---+---+------------------+
    //|1.0|3.0|               0.7|
    //|2.0|6.0|               1.4|
    //|3.0|9.0|2.0999999999999996|
    //+---+---+------------------+
    // only showing top 3 rows

With a normal Scala function there is no problem:

def foo(cols: Double*) = cols.reduce(​_+_​)

foo(1.2, 1.3, 1.6)

//res2: Double = 4.1

Although, when trying with a User Defined Function:

val fooudf = udf((cols: Double*) => cols.reduce(​_+_​))

//<console>:2: error: ')' expected but identifier found.
//      val fooudf = udf((cols: Double*) => cols.reduce(​_+_​))
//                                    ^

P.D. In python it is posible to do the task requested.

Hugo Reyes
  • 1,555
  • 2
  • 12
  • 21
  • For Spark UDF specific scenario see http://stackoverflow.com/q/33151866/1560062 – zero323 Apr 29 '16 at 16:36
  • Also UDF doesn't really require anonymous function so you could use: `udf(foo _)` although simple `def foo(cols: Column*) = cols.reduce(_ + _)` would make more sense for simple case like this. – zero323 Apr 29 '16 at 16:49

0 Answers0