Appending a new column to existing CSV file in Spark with Java

Question

I have found a solution to my problem here Create new column with function in Spark Dataframe

But i am having difficulty in converting the below code to Java since it's in Scala

import org.apache.spark.sql.functions._
val myDF = sqlContext.parquetFile("hdfs:/to/my/file.parquet")
val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}
val sqlfunc = udf(coder)
myDF.withColumn("Code", sqlfunc(col("Amt")))

Can someone provide me the Java equivalent code for this?. I am stuck in converting below 2 lines

val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}
val sqlfunc = udf(coder)

Thanks,

Well, clearly you would be able to convert some of this. So instead of asking for a code-translation service, please point out which particular bits are causing you a problem. — The Archetypal Paul, Feb 23 '16 at 08:14
@TheArchetypalPaul, Line 3 & 4 are troubling me, can you please provide me the translation for that?. Thanks — serverliving.com, Feb 23 '16 at 08:16

score 2 · Answer 1 · answered Feb 23 '16 at 14:53

Create your User Defined Function:

public class CodeUdf implements  UDF1<Integer, String>{
    @Override
    public String call(Integer integer) throws Exception {
        if(integer < 100)
            return "little";
        else
            return"big";
    }
}

Tell Spark about it

sqlContext.udf().register("Code", new CodeUdf(), DataTypes.IntegerType);

Use it in a select.

df.selectExpr("value", "Code(value)").show();

score -1 · Answer 2 · answered Feb 23 '16 at 11:55

import org.apache.spark.sql.functions._
val myDF = sqlContext.parquetFile("hdfs:/to/my/file.parquet")
//val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}
//val sqlfunc = udf(coder)
myDF.selectExpr("Code", "case when Amt < 100 'little' else 'big' end ")

Appending a new column to existing CSV file in Spark with Java

2 Answers2