1

I have a dataframe of the following format:

Name           Factor     Expression   Year 
Hydro          0.075            <10    2010  
Hydro          0.075            >10    2010  
Hydro          0.075            <10    2011  
Hydro          0.075            >10    2011
Hydro          0.075            <10    2012

And the following variable: i=3.

I would like to filter the dataframe where the Expression column evaluates as true, when the variable i is on the left hand side of the string in the column of expression.

For example, the first row would evaluate to true as 3<10.

The resulting dataframe that I would like is:

Name           Factor     Expression   Year 
Hydro          0.075            <10    2010  
Hydro          0.075            <10    2011  
Hydro          0.075            <10    2012

Thank you for any help.

AKE
  • 111
  • 1
  • 8

1 Answers1

2

Pandas has a safer version of eval that supports a limited number of operations. Luckily, > and < work, and you can use this along with string concatenation:

i = '3'
idx = pd.eval(i + df.Expression)
df.loc[idx]


    Name  Factor Expression  Year
0  Hydro   0.075        <10  2010
2  Hydro   0.075        <10  2011
4  Hydro   0.075        <10  2012

As @coldspeed noted, the above approach only works on DataFrames that are < 100 rows*, which isn't ideal. He also proposed the following solution:

df[[pd.eval(f"{i}{j}") for j in df['Expression']]]

*The above limitation is discussed more in depth in the following question: AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Ellipsis', using pandas eval

user3483203
  • 50,081
  • 9
  • 65
  • 94
  • Careful, `pd.eval` will only work if your DataFrame is 100 rows or smaller. – cs95 Dec 06 '18 at 20:30
  • Hmm, did not know that, really limits things quite a bit – user3483203 Dec 06 '18 at 20:31
  • 1
    If you want to generalise to any size, you should use `df[[pd.eval(f"{i}{j}") for j in df['Expression']]]` instead. – cs95 Dec 06 '18 at 20:31
  • @coldspeed do you know where that limitation is documented? I just get an attribute error when it reaches that size and I don't see it in the documentation – user3483203 Dec 06 '18 at 20:34
  • 1
    Yes! It is documented here: https://stackoverflow.com/questions/48008191/attributeerror-pandasexprvisitor-object-has-no-attribute-visit-ellipsis-us – cs95 Dec 06 '18 at 20:36