Access pandas masks in a dictionary

Question

I have a dictionary containing several pandas masks as strings for a specific dataframe, but I can't find a way to use those masks.

Here is a short reproducible example :

df = pd.DataFrame({'age' : [10, 24, 35, 67], 'strength' : [0 , 3, 9, 4]})

masks = {'old_strong' : "(df['age'] >18) & (df['strength'] >5)",
        'young_weak' : "(df['age'] <18) & (df['strength'] <5)"}

And I would like to do something like :

df[masks['young_weak']]

But since the mask is a string I get the error

KeyError: "(df['age'] <18) & (df['strength] <5)"

`df[eval(masks['young_weak'])]` ? - eval is usually frowned-upon... this would be a last resort probably — Chris Adams, Jun 05 '19 at 08:37

score 6 · Answer 1 · answered Jun 05 '19 at 08:30

6

Use DataFrame.query with changed dictionary:

masks = {'old_strong' : "(age >18) & (strength >5)",
        'young_weak' : "(age <18) & (strength <5)"}

print (df.query(masks['young_weak']))
   age  strength
0   10         0

answered Jun 05 '19 at 08:30

jezrael

822,522
95
1,334
1,252

Nice one ! But do you know if there is a way to do it without changing the dict, or by changing the dict to transform the strings into masks ? – vlemaistre Jun 05 '19 at 08:37
@vlemaistre - Unfortunately here is necessary change dictionary. – jezrael Jun 05 '19 at 08:38
Thanks for your answer @jezrael, I'll accept the one with eval() (even though it's less clean that your solution) because it was more of what I was looking for. But this helped me too :) – vlemaistre Jun 05 '19 at 08:45
@vlemaistre - yes, it is up to you - [Why is using 'eval' a bad practice?](https://stackoverflow.com/questions/1832940/why-is-using-eval-a-bad-practice) – jezrael Jun 05 '19 at 08:46
3

Just for reference.... `.query` is just using `pd.eval` under the hood anyway according to the [docs](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html)... – Chris Adams Jun 05 '19 at 08:46
@ChrisA - ya, but there are some check, because if change `eval` to `pd.eval` it failed – jezrael Jun 05 '19 at 08:47
@jezrael Your answer is much better that mine, but since his requirements only can be fulfilled with `eval`, i used it, but your answer is very good, you deserve the vote score you have now – U13-Forward Jun 05 '19 at 08:51
2

@U9-Forward - yop, maybe you can change your answer like it is posible, but strongly recomended dont do it. ;) – jezrael Jun 05 '19 at 08:53
1

@jezrael Added a little more to it – U13-Forward Jun 05 '19 at 08:55

score 1 · Answer 2 · answered Jun 05 '19 at 08:33

Another way is to set up the masks as functions (lambda expressions) instead of strings. This works:

masks = {'old_strong' : lambda row: (row['age'] >18) & (row['strength'] >5),
    'young_weak' :  lambda row: (row['age'] <18) & (row['strength'] <5)}
df[masks['young_weak']]

score 1 · Answer 3 · answered Nov 27 '22 at 17:18

If you're allowed to change the masks dictionary, the easiest way is to store filters and not strings like this:

masks = {
   'old_strong' : (df['age'] >18) & (df['strength'] >5),
   'young_weak' : (df['age'] <18) & (df['strength'] <5)
}

Otherwise, keep the strings and use df.query(masks['yound_weak']).

U13-Forward · Accepted Answer · 2019-06-05T08:54:47.213

0

Unsafe solution though, and very bad practice, but the only way to solve it is to use eval:

print(df[eval(masks['young_weak'])])

Output:

   age  strength
0   10         0

Here is the link to the reason it's bad.

edited Jun 05 '19 at 08:54

answered Jun 05 '19 at 08:38

U13-Forward

69,221
14
89
114

1

I saw this one in the comments, but I get an "EOL while scanning string literal" error when I try this solution – vlemaistre Jun 05 '19 at 08:39
2

your missing a closing `quote` after 'strength' in your dict - according to your `KeyError` above – Chris Adams Jun 05 '19 at 08:40
@vlemaistre Listen to Chris A – U13-Forward Jun 05 '19 at 08:40
1

Thanks that was it :) – vlemaistre Jun 05 '19 at 08:41

Access pandas masks in a dictionary

4 Answers4