2

Suppose i have a dataframe in which Timestamp column is present. 

Timestamp  
2016-04-19T17:13:17  
2016-04-20T11:31:31  
2016-04-20T18:44:31  
2016-04-20T14:44:01

I have to check whether current timsetamp is greater than Timestamp + 1 (i.e addition of 1 day to it) column in Scala

mrsrinivas
  • 34,112
  • 13
  • 125
  • 125
Tejas wagh
  • 19
  • 2
  • 11

2 Answers2

1

DataFrame supports two types of current_ on date and timestamp

Let's consider a DataFrame df with id and event_date columns.

We can perform the following filter operations :

import sqlContext.implicits._
import org.apache.spark.sql.functions._

// the event_date is before the current timestamp
df.filter('event_date.lt(current_timestamp()))

// the event_date is after the current timestamp
df.filter('event_date.gt(current_timestamp()))

I advice you to read the associated scala doc for more information here. You have a whole section on dates and timestamps operations.

EDIT: As discussed in the comments, in order to add a day to your event_date column, you can use the date_addfunction :

df.filter(date_add('event_date,1).lt(current_timestamp()))
nerdherd
  • 2,508
  • 2
  • 24
  • 40
eliasah
  • 39,588
  • 11
  • 124
  • 154
  • Thanks @eliasah it was helpful.. But suppose we want to filter 'event_date' + adding one more single day to it according to the date specified in column greater than current. Then how can we achieve it. – Tejas wagh May 02 '16 at 06:56
  • you can use date_add then it's also available in the doc. – eliasah May 02 '16 at 06:58
0

You can do it liked this.

df.filter(date_add('column_name', 1).lt(current_timestamp()))
Abhishek Anand
  • 1,940
  • 14
  • 27