4

I have an imported xls file as pandas dataframe, there are two columns containing coordinates which i will use to merge the dataframe with others which have geolocation data. df.info() shows 8859 records, the coordinatess columns have '8835 non-null float64' records.

I want to eyeball the 24 rows (that i assume are null) with all columns records to see if one of the other columns (street address town) can't be used to manually add back the coordinates for those 24 records. Ie. return dataframe for column in df.['Easting'] where isnull or NaN

I have adapted the method given here as below;

df.loc[df['Easting'] == NaN]

But get back an empty dataframe (0 rows × 24 columns), which makes no sense (to me). Attempting to use Null or Non null doesn't work as these values aren't defined. What am i missing?

Community
  • 1
  • 1
mapping dom
  • 1,737
  • 4
  • 27
  • 50

1 Answers1

8

I think you need isnull for checking NaN values with boolean indexing:

df[df['Easting'].isnull()]

Docs:

Warning

One has to be mindful that in python (and numpy), the nan's don’t compare equal, but None's do. Note that Pandas/numpy uses the fact that np.nan != np.nan, and treats None like np.nan.

In [11]: None == None
Out[11]: True

In [12]: np.nan == np.nan
Out[12]: False

So as compared to above, a scalar equality comparison versus a None/np.nan doesn’t provide useful information.

In [13]: df2['one'] == np.nan
Out[13]: 
a    False
b    False
c    False
d    False
e    False
f    False
g    False
h    False
Name: one, dtype: bool
Community
  • 1
  • 1
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
  • Thank you for the quick response, that worked a charm. I'm new to python and wasn't aware of the nan and none difference. – mapping dom Apr 24 '16 at 07:57
  • Glad can help you! Good luck! – jezrael Apr 24 '16 at 08:05
  • As an interesting aside to this i used this method to drop rows as part of the cleaning process as; `NLUDData_Clean=NLUDData.dropna(subset = ['Easting']) ` Even though I have fewer records the size as returned by .info() query shows memory usage to have increased 1.6+mb to 1.7+mb not an issue for me but interested to know why for future – mapping dom Apr 24 '16 at 08:55
  • Hard question for, maybe it is bug? – jezrael Apr 24 '16 at 09:07