1

Column1 = 43784.2892847338

Tried this below option

val finalDF=df1.withColumn("Column1 ",expr("""(Column1 -25569) * 86400.0"""))
print(finalDF)

Result : 1911120001 Expected result should be in "YYYY-MM-dd HH:mm:ss.SSS"

Could you please help me with the solution.

koiralo
  • 22,594
  • 6
  • 51
  • 72
Shobhana Mani
  • 43
  • 3

1 Answers1

0

If you use the from_unixtime then it will not give you the .SSS milliseconds. So, in this case, the casting is better to use.

val df1 = spark.createDataFrame(Seq(("1", 43784.2892847338))).toDF("id", "Column1")

val finalDF = df1.withColumn("Column1_timestamp", expr("""(Column1 -25569) * 86400.0""").cast("timestamp"))
finalDF.show(false)

+---+----------------+-----------------------+
|id |Column1         |Column1_timestamp      |
+---+----------------+-----------------------+
|1  |43784.2892847338|2019-11-15 06:56:34.201|
+---+----------------+-----------------------+
Lamanus
  • 12,898
  • 4
  • 21
  • 47