spark scala how can I calculate days since 1970-01-01

Question

Looking for scala code to replicate https://www.epochconverter.com/seconds-days-since-y0

I have a spark streaming job reading the avro message. The message has a column of type int and holds Days Since 1970-01-01. I want to convert that to date.

 dataFrame.select(from_avro(col("Value"), valueRegistryConfig) as 'value)
 .select("value.*")
 .withColumn("start_date",'start_date)

start_date is holding an integer value like 18022 i.e Days Since 1970-01-01. I want to convert this value to a date

18022 - > Sun May 05 2019

Does this answer your question? [Add Number of days column to Date Column in same dataframe for Spark Scala App](https://stackoverflow.com/questions/44361332/add-number-of-days-column-to-date-column-in-same-dataframe-for-spark-scala-app) — Fang Zhang, Dec 18 '19 at 18:30

score 2 · Accepted Answer · answered Dec 18 '19 at 18:30

Use default date as 1970-01-01 and pass number of days to date_add function.

This will give you date but will be 1 day additional so you do minus 1.

Something like this:

var dataDF = Seq(("1970-01-01",18091),("1970-01-01",18021),("1970-01-01",18022)).toDF("date","num")
dataDF.select(
    col("date"),
    expr("date_add(date,num-1)").as("date_add")).show(10,false)

+----------+----------+
|date      |date_add  |
+----------+----------+
|1970-01-01|2019-07-13|
|1970-01-01|2019-05-04|
|1970-01-01|2019-05-05|
+----------+----------+

spark scala how can I calculate days since 1970-01-01

1 Answers1