pyspark sql dataframe keep only null

Question

I have a sql dataframe df and there is a column user_id, how do I filter the dataframe and keep only user_id is actually null for further analysis? From the pyspark module page here, one can drop na rows easily but did not say how to do the opposite.

Tried df.filter(df.user_id == 'null'), but the result is 0 column. Maybe it is looking for a string "null". Also df.filter(df.user_id == null) won't work as it is looking for a variable named 'null'

score 5 · Accepted Answer · edited Jun 15 '16 at 18:37

5

Try

df.filter(df.user_id.isNull())

edited Jun 15 '16 at 18:37

zero323

322,348
103
959
935

answered Jun 14 '16 at 21:16

David

11,245
3
41
46

pyspark sql dataframe keep only null

1 Answers1