How to implement EXISTS condition as like SQL in spark Dataframe

Question

I am curious to know, how can i implement sql like exists clause in spark Dataframe way.

Possible duplicate of [Spark replacement for EXISTS and IN](https://stackoverflow.com/questions/34861516/spark-replacement-for-exists-and-in) — pault, Jan 08 '20 at 15:35

score 4 · Accepted Answer · answered Jan 07 '20 at 11:24

4

LEFT SEMI JOIN is equivalent to the EXISTS function in Spark.

val cityDF= Seq(("Delhi","India"),("Kolkata","India"),("Mumbai","India"),("Nairobi","Kenya"),("Colombo","Srilanka")).toDF("City","Country")

val CodeDF= Seq(("011","Delhi"),("022","Mumbai"),("033","Kolkata"),("044","Chennai")).toDF("Code","City")

val finalDF= cityDF.join(CodeDF, cityDF("City") === CodeDF("City"), "left_semi")

answered Jan 07 '20 at 11:24

venus

Any other way instead of join operation? Since I have too many joins in my Sql query and I don't want to make it more complex using this join syntax. Any other way will be appreciated – Sagar patro Jan 07 '20 at 11:27
sorry bro :(.. this is what I know..post another question with good explanation. – venus Jan 07 '20 at 11:36

score 1 · Answer 2 · edited Aug 22 '21 at 18:46

1

If the data to be compared is small like a broadcasted list then you can use -

df.filter(col("columnName").isin(list...))

edited Aug 22 '21 at 18:46

Topde

answered Jan 07 '20 at 23:44

Salim

2 Answers2