Dataframe empty check pyspark

Question

I am trying to check if a dataframe is empty in Pyspark using below.

print(df.head(1).isEmpty)

But, I am getting an error

Attribute error: 'list' object has no attribute 'isEmpty'.

I checked if my object is really a dd using type(df) and it is class 'pyspark.sql.dataframe.Dataframe'

Does this answer your question? [How to check if spark dataframe is empty?](https://stackoverflow.com/questions/32707620/how-to-check-if-spark-dataframe-is-empty) — ZygD, Sep 17 '21 at 11:15

score 2 · Accepted Answer · answered Sep 17 '21 at 11:16

2

I used df.first() == None to evaluate if my spark dataframe is empty

answered Sep 17 '21 at 11:16

Padfoot123

1

`df.first() is None` would be the preferred syntax. – Nikhil Gupta Jun 06 '22 at 20:17

score 1 · Answer 2 · edited Sep 17 '21 at 11:16

1

When u do a head(1) it returns a list. So that’s the reason for your error.

You have to just do df.isEmpty().

edited Sep 17 '21 at 11:16

ZygD

answered Sep 17 '21 at 09:33

Learner

Now getting dataframe object has no attribute isEmpty – Padfoot123 Sep 17 '21 at 09:35
1

Df.first() == None worked for me – Padfoot123 Sep 17 '21 at 09:40

score 0 · Answer 3 · edited May 11 '22 at 10:14

0

df.head(1) returns a list corresponding to the first row of df.

You can check if this list is empty "[ ]" using a bool type condition as in:

if df.head(1):
    print("there is something")
else:
    print("df is empty")

>>> 'df is empty'

Empty lists are implicity "False".

For better explanation, please head over to python docs.

edited May 11 '22 at 10:14

Andronicus

answered May 10 '22 at 16:52

andrelsjunior

score 0 · Answer 4 · answered May 11 '22 at 12:27

0

Another way to do this would be to check if df.count()==0

answered May 11 '22 at 12:27

fuzzy-memory

This could cause unnecessary computation on Spark, especially for large datasets. – user1234 Feb 16 '23 at 11:57

score 0 · Answer 5 · answered Aug 29 '23 at 14:58

0

Df.isEmpty would work for everyone starting from spark version 3.3.0. Earlier spark version might not support isEmpty method for pyspark.

answered Aug 29 '23 at 14:58

Nikunj Kakadiya

5 Answers5