java, how to call UDF in spark 1.4.1

Question

In spark 1.4.1, the parameter of callUdf method is

(String udfName, scala.collection.Seq<Column> cols)

There is no method that can act to column directly, like method in 1.5.1

callUDF(String udfName, Column col)

So how to call UDF in 1.4.1? Or how to change column type to

scala.collection.Seq<Column>

For example, these codes work in 1.6.1

sqlContext.udf().register("stringToLong", new UDF1<String, Long>() {

    @Override
    public Long call(String arg0) throws Exception {
        // TODO Auto-generated method stub
        IPTypeConvert itc = new IPTypeConvert();
        return itc.stringtoLong(arg0);
    }
}, DataTypes.LongType);
DataFrame interDF = initInterDF.withColumn("interIPInt", callUDF("stringToLong", initInterDF.col("interIP")));

How should I change the codes so that they can work in spark 1.4.1?

score 1 · Accepted Answer · edited May 23 '17 at 12:07

1

You can convert list to Seq[Column]:

import scala.collection.JavaConversions;
import scala.collection.Seq;

import static java.util.Arrays.asList;

DataFrame interDF = initInterDF.withColumn("interIPInt", 
    callUDF("stringToLong", JavaConversions.asScalaBuffer(asList(initInterDF.col("interIP")))));

See more in this question

edited May 23 '17 at 12:07

Community

1
1

answered Nov 28 '16 at 13:54

T. Gawęda

15,706
4
46
61

Forgive me, I am a rookie. Thank you very much! – volity Nov 28 '16 at 14:41
@volity Relax :) I'm happy I have helped you :) – T. Gawęda Nov 28 '16 at 14:45

java, how to call UDF in spark 1.4.1

1 Answers1