0

I've a DataFrame with a TimestampType column, I'm reading the data manually then constructing the DataFrame. In the input, the original DateTime column has timezone information, e.g. 2011-11-04T00:05:23+04:00 Now when I read the data in a Spark Timestamp column I realized that the timezone is gone!

This is how I am constructing the schema for my DataFrame

var fields = ...
fields = fields :+ StructField("timestamp", TimestampType, false)
val schema StructType(fields)

And how I parse the dates into a java.sql.Timestamp

val date = new Timestamp(x)

I've end up adding a separate column that contains the TimeZone, but is there a better option (other than making the column a StringType and serializing the original date)?

bachr
  • 5,780
  • 12
  • 57
  • 92
  • 1
    I don't think it's possible. See: https://stackoverflow.com/questions/14070572/is-java-sql-timestamp-timezone-specific – moon Apr 30 '20 at 01:47
  • Yeah i'm unfortunately I aware of this TimestampType limitation, was just wondering if there is any workarounds. – bachr Apr 30 '20 at 06:33

0 Answers0