1

I'm trying to make a statement that basically says if EDW_ABC.edw_xdpt_act_arrv_lb is NULL then use EDW_ABC.edw_putt_act_arrv_lb and if both are null then set the value to 0. How do I do that? I'm trying the below and I know it's not correct.

EDW_ABC = EDW_ABC.withColumn('act_arrv_abc_lbs', F.when(
(EDW_ABC.edw_xdpt_act_arrv_lb.isNull() == True) & (EDW_ABC.edw_putt_act_arrv_lb.isNull() == True). F.lit(0)\
                                         .otherwise(EDW_ABC.edw_xdpt_act_arrv_lb.isNull()), EDW_ABC.edw_putt_act_arrv_lb)
Adil B
  • 14,635
  • 11
  • 60
  • 78
confused101
  • 99
  • 1
  • 7
  • Semi duplicate of, find more details there: https://stackoverflow.com/questions/39048229/spark-equivalent-of-if-then-else – ollie299792458 Oct 26 '21 at 09:47
  • Does this answer your question? [Spark Equivalent of IF Then ELSE](https://stackoverflow.com/questions/39048229/spark-equivalent-of-if-then-else) – ollie299792458 Oct 27 '21 at 11:46

2 Answers2

2

You will want to use the pyspark.sql.functions.coalesce() function, which returns the value from the first column in the list that does not return NULL.

EDW_ABC = EDW_ABC.withColumn('act_arrv_abc_lbs', F.coalesce(F.col("edw_xdpt_act_arrv_lb"), F.col("edw_putt_act_arrv_lb"), F.lit(0))
Kate
  • 86
  • 4
1

You don't need to specify a condition in the otherwise, so

EDW_ABC = EDW_ABC.withColumn(
    'act_arrv_abc_lbs',
    F.when(
        EDW_ABC.edw_xdpt_act_arrv_lb.isNull() & EDW_ABC.edw_putt_act_arrv_lb.isNull(), F.lit(0)
    ).otherwise(
        EDW_ABC.edw_putt_act_arrv_lb
    )
)

when and otherwise operate as if & else, so if the first condition in the when isn't satisfied, the otherwise automatically assumes the opposite.

youssefrizk
  • 129
  • 3