What is the use of lit() in spark? The below two piece of code returns the same output, what is the benefit of using lit()

Question

I have two piece of codes here

gooddata = gooddata.withColumn("Priority",when(gooddata.years_left < 5 & (gooddata.Years_left >= 0),lit("CRITICAL"))).fillna("LOW").show(5)

gooddata=gooddata.withColumn("Priority",when((gooddata.Years_left < 5) & (gooddata.Years_left >= 0),"CRITICAL").otherwise("LOW")).show(5)

Does this answer your question? [Where do you need to use lit() in Pyspark SQL?](https://stackoverflow.com/questions/37715060/where-do-you-need-to-use-lit-in-pyspark-sql) — user10938362, Jun 09 '20 at 17:59

score 1 · Answer 1 · answered Jun 09 '20 at 17:39

For both spark and pyspark:

literals in certain statements
comparing with nulls
getting the name of a dataframe column instead of the contents of the dataframe column

E.g.

val nonNulls = df.columns.map(x => when(col(x).isNotNull, concat(lit(","), lit(x))).otherwise(",")).reduce(concat(_, _))

from question: Add a column to spark dataframe which contains list of all column names of the current row whose value is not null

val df2 = df.select(col("EmpId"),col("Salary"),lit("1").as("lit_value1"))

What is the use of lit() in spark? The below two piece of code returns the same output, what is the benefit of using lit()

1 Answers1