Filter out dictionary with nan values

Question

Let's say I have a dictionary:

dict_1 = {
    "key_1": [0., 5.6, 6.1, np.nan],
    "key_2": ["a", "t", "g", "r"],
    "key_3": [6.7, np.nan, 5.6, 4.1]
}

All keys have values that are list and they all have the same length. I want to filter out the elements that are np.nan (including the elements in the other keys that have the same index), so this is the desired output:

result = {
    "key_1": [0., 6.1],
    "key_2": ["a", "g"],
    "key_3": [6.7, 5.6]
}

Can someone please help? I don't want to do for loop because that's slow. I tried np.isnan(task_dictionary.values()).any(axis=1) but it failed because key_2 element type is string.

Thanks!

I'm pretty sure you'll have to loop over the dictionary yourself, even if numpy has some magic that can filter the lists for you — Anentropic, May 19 '22 at 18:38
basically i'm thinking iterating through the elements is not efficient because the dictionary can contain > 100k elements — eng2019, May 19 '22 at 18:39
You can try pandas and make that dict (assuming in reality would be large) into dataframe remove and return back to dict — coldy, May 19 '22 at 18:40

Jab · Answer 1 · 2022-05-19T19:32:35.557

You can use Pandas DataFrame.dropna:

pd.DataFrame(dict_1).dropna().to_dict('list')

Result:

{'key_1': [0.0, 6.1], 'key_2': ['a', 'g'], 'key_3': [6.7, 5.6]}

Full Code (breakdown):

import pandas as pd
import numpy as np

dict_1 = {'key_1': [0.0, 5.6, 6.1, np.nan], 'key_2': ['a', 't', 'g', 'r'], 'key_3': [6.7, np.nan, 5.6, 4.1]}

df = pd.DataFrame(dict_1)
# key_1 key_2  key_3
# 0    0.0     a    6.7
# 1    5.6     t    NaN
# 2    6.1     g    5.6
# 3    NaN     r    4.1

df.dropna(inplace=True)
# key_1 key_2  key_3
# 0    0.0     a    6.7
# 2    6.1     g    5.6

df.to_dict('list')
# {'key_1': [0.0, 6.1], 'key_2': ['a', 'g'], 'key_3': [6.7, 5.6]}

Without pandas, I believe your most efficient approach would be to use this mess using zip:

dict(zip(dict_1, zip(*(e for e in zip(*dict_1.values()) if np.nan not in e))))

BrokenBenchmark · Answer 2 · 2022-05-19T18:44:43.443

If you don't want to use any dependencies, you can first find all of the column indices to keep using this transposition trick and a set comprehension that checks whether a given column has any entries that aren't equal to themselves (since NaN != NaN). Then, use a dictionary comprehension to retain only those values:

indices_to_keep = {i for i, entry in enumerate(zip(*dict_1.values())) if all(x == x for x in entry)}
{k: [x for i, x in enumerate(v) if i in indices_to_keep] for k, v in dict_1.items()}

This outputs:

{'key_1': [0.0, 6.1], 'key_2': ['a', 'g'], 'key_3': [6.7, 5.6]}

explodingfilms101 · Answer 3 · 2022-05-19T19:21:37.567

0

There's no way to do this without looping. You could make it a bit faster than regular for loops by using list comprehension though:


result = {i:dict_1[i] for i in dict_1 if np.nan not in dict_1[i]}

edited May 19 '22 at 19:21

answered May 19 '22 at 18:44

explodingfilms101

480
2
11
23

what is `a` supposed to be? – Jab May 19 '22 at 18:57
@Jab My apologies, I was testing this and forgot to change all the variables to fit your question. – explodingfilms101 May 19 '22 at 19:00
You have an extra closing bracket in there causing an error and even fixing that, this code doesn't even get expected results. This results in: `{'key_2': ['a', 't', 'g', 'r']}` – Jab May 19 '22 at 19:20

Filter out dictionary with nan values

3 Answers3