14

I have a pandas DataFrame that I want to separate into observations for which there are no missing values and observations with missing values. I can use dropna() to get rows without missing values. Is there any analog to get rows with missing values? 

#Example DataFrame
import pandas as pd
df = pd.DataFrame({'col1': [1,np.nan,3,4,5],'col2': [6,7,np.nan,9,10],})

#Get observations without missing values
df.dropna()
Gaurav Bansal
  • 5,221
  • 14
  • 45
  • 91

3 Answers3

31

Check null by row and filter with boolean indexing:

df[df.isnull().any(1)]

#  col1 col2
#1  NaN  7.0
#2  3.0  NaN
Psidom
  • 209,562
  • 33
  • 339
  • 356
  • 4
    Or if you were super concerned about performance, `df[np.isnan(df.values).any(1)]` - the difference in `any`'s performance between an ndarray and a DataFrame has always seemed noticeable to me. – miradulo Oct 08 '17 at 01:46
8

~ = Opposite :-)

df.loc[~df.index.isin(df.dropna().index)]

Out[234]: 
   col1  col2
1   NaN   7.0
2   3.0   NaN

Or 

df.loc[df.index.difference(df.dropna().index)]
Out[235]: 
   col1  col2
1   NaN   7.0
2   3.0   NaN
BENY
  • 317,841
  • 20
  • 164
  • 234
2

I use the following expression as the opposite of dropna. In this case, it keeps rows based on the specified column that are null. Anything with a value is not kept.

csv_df = csv_df.loc[~csv_df['Column_name'].notna(), :]
PhilipBert
  • 25
  • 1
  • 3