Passing List as argument in spark scala function gives error

Question

I have one spark scala udf which takes one parameter as column of dataframe and other parameter as List but when I run the function it throws error pointing to list argument as

type mismatch, found spark.sql.row ,required spark.sql.column

I am running udf with argument as follows,

udf_name($"column_name",List_name)

Please Guide

Answer 1

You need to define multiple instances of your UDF with the lists you want to pass. Since the lists are local scala variables you can do that just before the call (spark will ship the udf to the various executors) e.g.

import org.apache.spark.sql.functions._
val df=List("A","B").toDF
def to_be_udf(s: String, l : List[String])=if (l.isEmpty) "" else "has values"
val udf1=udf((s:String) => to_be_udf (s,List("a")))
val udf2=udf((s:String) => to_be_udf (s,List()))
df.select(udf1($"value"),udf2($"value")).show()

+----------+----------+
|UDF(value)|UDF(value)|
+----------+----------+
|has values|          |
|has values|          |
+----------+----------+

Answer 2

You can either pass the constant value to an udf using lit, or alternatively define a method with returns an UDF (my preferred way):

def udf_name(List_name:List[String]) = {
  udf((name:String) => {
    // do something 
    List_name.contains(name)
  })
}

val List_name : List[String] = ???

df
  .withColumn("is_name_in_list", udf_name(List_name)($"column_name"))