4

I am trying to remove keys with nan values from a dictionary formed from pandas using python. Is there a way I can achieve this. Here is a sample of my dictionary:

{'id': 1, 'internal_id': '1904', 'first_scraping_time': '2020-04-17 12:44:59.0', 'first_scraping_date': '2020-04-17', 'last_scraping_time': '2020-06-20 03:08:47.0', 'last_scraping_date': '2020-06-20', 'is_active': 1,'flags': nan, 'phone': nan,'size': 60.0, 'available': '20-06-2020', 'timeframe': nan, 'teaser': nan, 'remarks': nan, 'rent': 4984.0, 'rooms': '3', 'downpayment': nan, 'deposit': '14952', 'expenses': 600.0, 'expenses_tv': nan, 'expenses_improvements': nan, 'expenses_misc': nan, 'prepaid_rent': '4984', 'pets': nan, 'furnished': nan, 'residence_duty': nan, 'precision': nan, 'nearby_cities': nan,'type_dwelling': nan, 'type_tenants': nan, 'task_id': '614b8fc2-409c-403a-9650-05939e8a89c7'}

Thank you!

Chris_Rands
  • 38,994
  • 14
  • 83
  • 119
Derrick Omanwa
  • 457
  • 1
  • 5
  • 23
  • `{k : v for k,v in d.items() if v is not nan}` ? where d is your dictionary then re-assign to a new variable. set `nan` to `np.nan` i.e `nan = np.nan` – Umar.H Jun 23 '20 at 12:31
  • 2
    @Datanovice `nan is nan` is not necessarily true. – L3viathan Jun 23 '20 at 12:33
  • @L3viathan if its coming from a pandas function won't it be equal to `numpy.nan` rather than `math.isnan` regardless very helpful post thank you – Umar.H Jun 23 '20 at 12:37
  • 2
    Does this answer your question? [How can I check for NaN values?](https://stackoverflow.com/questions/944700/how-can-i-check-for-nan-values) – Chris_Rands Jun 23 '20 at 12:47

1 Answers1

6

nan is a tricky object to work with because it doesn't equal (or even necessarily share object identity) with anything, including itself.

You can use math.isnan to test for it:

import math
new = {key: value for (key, value) in old.items() if not math.isnan(value)}
L3viathan
  • 26,748
  • 2
  • 58
  • 81
  • 3
    `pd.isna` can be used instead of `math.isnan`, posting this comment 'cause pandas is tagged. `pd.isna(math.nan)`-> `True`, `pd.isna(np.nan)`->`True`. +1 – Ch3steR Jun 23 '20 at 12:40
  • I noticed that both pandas and numpy have such functions. Is there any benefit, apart from being able to apply them to arrays/matrices? – L3viathan Jun 23 '20 at 12:50
  • `pd.isna` can handle `None` too, while `math.isnan` and `np.isnan` can't. And like you mentioned `pd.isna` and `np.isnan` can applied be on arrays. – Ch3steR Jun 23 '20 at 12:56
  • In that case, I think `math.isnan` is a better answer. `nan` is not the same as `None` (which OP may _not_ want to filter out), and it's being used on a single element here. – L3viathan Jun 23 '20 at 13:50