isNullOrEmpty function in spark to check column in data frame is null or empty string

Question

How can I check the columns of dataframe is null or empty ins spark.

Ex.

type IdentifiedDataFrame = {SourceIdentfier, DataFrame}
def splitRequestIntoDFsWithAndWithoutTransactionId(df: DataFrame) : Seq[IdentifiedDataFrame] = {
  seq((DeltaTableStream(RequestWithTransactionId), df.filter(col(RequestLocationCodeColName).isNull
      && col(ServiceNumberColName).isNull
      && col(DateOfServiceColName).isNull
      && col(TransactionIdColName).isNotNull)).
    
      (DeltaTableStream(RequestWithoutTransactionId), df.filter(col(RequestLocationCodeColName).isNotNull
       && col(ServiceNumberColName).isNotNull
       && col(DateOfServiceColName).isNotNull))
      )
}

Note : this code only check the null value in column and I want to check null or empty string both Please help

score 1 · Answer 1 · answered Jul 08 '21 at 09:39

1

You can use isNull function and check for empty String with filter as below

val columns = List("column1", "column2")

val filter = columns.map(c => isnull(col(c)) || !(col(c) <=> lit("")))
  .reduce(_ and _)

df.filter(filter)

answered Jul 08 '21 at 09:39

koiralo

22,594
6
51
72

Hi Koirala, i paseted the main code where is am using isNull but now i have to make function and use isNullOrEmpty instead of isNull can you please help in this – Aditya Mishra Jul 08 '21 at 13:28
But this was not your original question, and title doesnot match the contents – koiralo Jul 08 '21 at 13:41
Ohh can u pls help me in this – Aditya Mishra Jul 08 '21 at 14:14
In place of isNull i want to use isNullOrEmpty and inplace of isNotNull I want use isNotNullOrEmpty – Aditya Mishra Jul 08 '21 at 14:16
I am new in development and I need to do this. It will very helpful if u help me out – Aditya Mishra Jul 08 '21 at 14:17
The question is not just clear to me , take a look here if this helps https://stackoverflow.com/questions/60213412/spark-scala-check-if-string-isnt-null-or-empty – koiralo Jul 08 '21 at 14:25
Thank you, same thing I also want to do. But the link provided by you giving me type mismatch error – Aditya Mishra Jul 08 '21 at 14:27
I just want to make a function similar to isNull but it should also check empty string with null value. – Aditya Mishra Jul 08 '21 at 14:28

isNullOrEmpty function in spark to check column in data frame is null or empty string

1 Answers1

Linked