How to create a new column based on values of two columns in Spark

Question

So, I want to create a new column in my dataframe, whose rows depend upon values from two columns, and also involves a condition.

I tried this, but it doesn't work.

some_value = ...
df = df.withColumn("new_col", col("col1") if col("col2") == some_value else None)

What is the correct way of doing this?

you can take a look at `when` and `otherwise` , if that doesnt help , please create a testable example — anky, Apr 29 '20 at 17:22
Does this answer your question? [PySpark: Create New Column And Fill In Based on Conditions of Two Other Columns](https://stackoverflow.com/questions/51565395/pyspark-create-new-column-and-fill-in-based-on-conditions-of-two-other-columns) What happen to google?! — mazaneicha, Apr 29 '20 at 17:31

score 0 · Answer 1 · answered Apr 29 '20 at 17:33

0

A trivial example using with expr, can also use when:

val df3 = df2.withColumn("new_col", expr("case when c1 = 1 and c2 = 101 then c1 + c2 else 999 end"))

answered Apr 29 '20 at 17:33

thebluephantom

score 0 · Answer 2 · answered Apr 29 '20 at 17:34

0

This worked for me,

 new_col_expr = when(col("col2").eqNullSafe(some_value), col("col1")).otherwise(None)
 df = df.withColumn("new_col", new_col_expr)

answered Apr 29 '20 at 17:34

MetallicPriest

score 0 · Accepted Answer · answered Apr 29 '20 at 17:38

0

You can use when function.

newval='10002'
df2 = df.withColumn("new_col", when((col("col1") == lit(newval)), "col1").otherwise(None))
df2.show()

answered Apr 29 '20 at 17:38

Piyush Patel

What do I need to import to use lit? – MetallicPriest Apr 29 '20 at 17:44
1

Not sure if you imported, you can use `import org.apache.spark.sql.functions._` to use `lit` – Piyush Patel Apr 30 '20 at 02:05

3 Answers3