Add date in pyspark data frame

Question

I'm using Azure databricks Runtime 7.3 LTS I have pyspark dataframe df. In df I want to add a new column Date_time which will have date value. So I wrote following comments

from pyspark.sql import functions as F
timestamp='2020-01-03'
df.withColumn('Date_time', timestamp)

But I'm getting error message

AssertionError: col should be Column

How to rectify this error?

Does this answer your question? [How to add a constant column in a Spark DataFrame?](https://stackoverflow.com/questions/32788322/how-to-add-a-constant-column-in-a-spark-dataframe) — mazaneicha, Mar 23 '21 at 14:28

score 1 · Answer 1 · answered Nov 26 '20 at 07:23

if you want your 'Date_time' column to have literal value then you can use lit function for this.

you need to modify your code as follows :

from pyspark.sql import functions as F
timestamp='2020-01-03'
df = df.withColumn('Date_time', F.lit(timestamp))

score 1 · Accepted Answer · answered Nov 26 '20 at 08:45

Adding to the other answer, you might also want to cast the column to timestamp type or date type. Otherwise by default it will be of string type, which is probably not what you want.

import pyspark.sql.functions as F

timestamp = '2020-01-03'
df = df.withColumn('Date_time', F.lit(timestamp).cast('timestamp'))

# or, if you want a date type instead,
# df = df.withColumn('Date_time', F.lit(timestamp).cast('date'))

Add date in pyspark data frame

2 Answers2