-2

I am using below code but it gives error.Kindly guide.

val a = Seq(
  ("ram,shyam,hari",12,10),
  ("shyam,ram,hari",3,5)
).toDF("name","id","dt")
  .withColumn("newcol",if($"id">$"dt",0,1))
  .show()

Error is as follows,

:14: error: ')' expected but ',' found. .withColumn("newcol",if($"id">$"dt",0,1)).show()

Tzach Zohar
  • 37,442
  • 3
  • 79
  • 85
user8167344
  • 353
  • 2
  • 6
  • 17

2 Answers2

1

You need when.otherwise:

val df = Seq(("ram,shyam,hari",12,10),("shyam,ram,hari",3,5)).toDF("name","id","dt")

df.withColumn("newcol", when($"id" > $"dt", 0).otherwise(1)).show
//+--------------+---+---+------+
//|          name| id| dt|newcol|
//+--------------+---+---+------+
//|ram,shyam,hari| 12| 10|     0|
//|shyam,ram,hari|  3|  5|     1|
//+--------------+---+---+------+

Or you can cast the comparison result to int:

df.withColumn("newcol", ($"id" <= $"dt").cast("int")).show
//+--------------+---+---+------+
//|          name| id| dt|newcol|
//+--------------+---+---+------+
//|ram,shyam,hari| 12| 10|     0|
//|shyam,ram,hari|  3|  5|     1|
//+--------------+---+---+------+
Psidom
  • 209,562
  • 33
  • 339
  • 356
0

Use when / otherwis:

import org.apache.spark.sql.functions.when

df.withColumn("newcol", when($"id" > $"dt", 0).otherwise(1))
user10230145
  • 1
  • Dear All thank you for your answers,actually I know how to achieve by other methods.My question is why it is not done by specifically by 'if' function which is provided by spark. – user8167344 Aug 16 '18 at 13:27