Scala Spark : Convert Double Column to Date Time Column in dataframe

Question

I am trying to write code to convert date-time columns date and last_updated_date which are actually unix times cast as doubles into "mm-dd-yyyy" format for display. How do I do this ?

import org.joda.time._
import scala.tools._
import org.joda.time.format.DateTimeFormat._
import java.text.SimpleDateFormat
import org.apache.spark.sql.functions.{unix_timestamp, to_date}
root
 |-- date: double (nullable = false)
 |-- last_updated_date: double (nullable = false)
 |-- Percent_Used: double (nullable = false)

+------------+---------------------+------------+
|        date|    last_updated_date|Percent_Used|
+------------+---------------------+------------+
| 1.453923E12|        1.47080394E12| 1.948327124|
|1.4539233E12|        1.47080394E12| 2.019636442|
|1.4539236E12|        1.47080394E12| 1.995299371|
+------------+---------------------+------------+

Did you figure it out? If so can you help me with the answer — Johny_Bravo, Apr 02 '19 at 06:48

score 6 · Accepted Answer · answered Aug 29 '16 at 16:27

6

Cast to timestamp:

df.select(col("date").cast("timestamp"));

answered Aug 29 '16 at 16:27

Hmm i think there is some issue with this since date is "long" and that is causing the dates to go out of whack when doing this directly – Leothorn Aug 29 '16 at 22:37

score 1 · Answer 2 · answered Aug 30 '16 at 00:55

1

Convert it to a timestamp using from_unixtime:

df.select(from_unixtime("date").as("date"))

answered Aug 30 '16 at 00:55

Justin Pihony

66,056
18
147
180

score 1 · Answer 3 · answered Jan 27 '20 at 19:58

Fetching datetime from float in Python

This answer works for me give a try actually its a seconds calculation

import datetime serial = 43822.59722222222 seconds = (serial - 25569) * 86400.0 print(datetime.datetime.utcfromtimestamp(seconds))

Convert excel timestamp double value into datetime or timestamp

Scala Spark : Convert Double Column to Date Time Column in dataframe

3 Answers3

Linked