Python: fast functions from string evaluations?

Question

I might tackling this completely wrong, but I am curious if this can be done in Python.

I am trying to built a function that takes a string and returns a function based on that string. For instance, given b*exp(a*x) and a list of inputs ['a','b','c'] is there a way to create this function dynamically in Python?

def f_fast(a, b, x):
    return b*np.exp(a*x)

I can see how I could create a slow version of that using eval:

np_funcs = {'exp':np.exp, 'sin':np.sin, 'cos':np.cos}

def make_func(s, vars):
    def f(*x):
        d = {e:x[i] for i, e in enumerate(vars)}
        values = dict(d.items() + np_funcs.items())
        return eval(s, {"__builtins__": None}, values)
    return f

s = 'b*exp(a*x)'
f = make_func(s, ['a', 'b', 'x'])

But this function will do string evaluation of every call. I wonder if there is a way to do the translation of string into functions only at creation and then subsequent calls will be fast.

Currently this implementation is very slow:

x = np.linspace(0,1,10)
print timeit.timeit('f(1,2,x)', "from __main__ import f, x, f_fast", number=10000)
print timeit.timeit('f_fast(1,2,x)', "from __main__ import f, x, f_fast", number=10000)

returns

0.16757759497
0.0262638996569

Any help, including explaining why this can't be done or why it is a stupid approach, would be greatly appreciated.

Thank you in advance.

you have to create a grammar and then parse the expression. fast is relative... what you want to do with the tree built from the expression? just evaluate it? JIT compile it? — Karoly Horvath, Apr 08 '17 at 18:46
If you want a one-liner, just use `lambda a, b, x: b*np.exp(a*x)` — gyre, Apr 08 '17 at 18:46
[`{'__builtins__': None}` will not save you](http://stackoverflow.com/questions/2371436/evaluating-a-mathematical-expression-in-a-string#comment39686914_2372145). — Ilja Everilä, Apr 08 '17 at 18:47
@gyre Doesn't compile in Python 3: `lambda` syntax takes no parentheses around arguments. — Right leg, Apr 08 '17 at 18:49
@KarolyHorvath: Just evaluate it, yes. But I need to do this many times, so I need it to be fast. — Julius, Apr 08 '17 at 18:50
You should take a look at [this post](http://stackoverflow.com/questions/2371436/evaluating-a-mathematical-expression-in-a-string) for pointers. — Ilja Everilä, Apr 08 '17 at 18:51
@gyre: this does not solve the issue at all. You are given a string (e.g. from a web input field or something) and need to be able to make fast calls to that string. — Julius, Apr 08 '17 at 18:52
@Julius You never mentioned your reasoning for needing a string in the question, so I thought it was worth a shot. At least now we all know why you turned to `eval`. — gyre, Apr 08 '17 at 18:54
you could build `f=eval('lambda a,b,x: b*np.exp(a*x)')` and it would be fast to evaluate, but it's still evil ;) — Karoly Horvath, Apr 08 '17 at 18:58

score 2 · Accepted Answer · answered Apr 08 '17 at 19:10

The safe way to do this is a pain. You'd parse the expression, and then emit an AST which you can pass to compile(). Creating the AST looks like this:

import ast
expr = ast.Expression(
    lineno=1,
    body=ast.Lambda(
        args=ast.arguments(
            args=[
                ast.arg(arg='a'),
                ast.arg(arg='b'),
                ast.arg(arg='x'),
            ],
            kwonlyargs=[],
            kw_defaults=[],
            defaults=[],
        ),
        body=ast.BinOp(
            left=ast.Name(id='b', ctx=ast.Load()),
            op=ast.Mult(),
            right=ast.Call(
                func=ast.Attribute(
                    value=ast.Name(id='np', ctx=ast.Load()),
                    attr='exp',
                    ctx=ast.Load(),
                ),
                args=[ast.BinOp(
                    left=ast.Name(id='a', ctx=ast.Load()),
                    op=ast.Mult(),
                    right=ast.Name(id='x', ctx=ast.Load()),
                )],
                keywords=[],
            ),
        ),
    ),
)
ast.fix_missing_locations(expr)

You can then turn it into an optimized function and call it:

code = compile(expr, 'myinput', 'eval', optimize=1)
func = eval(code, {'np': np})
print(func(1, 2, 3))

The task of creating the AST is the hard part. You can either create it yourself, like above, or you can pass the flag ast.PyCF_ONLY_AST to compile() and then sanitize the tree... but the difficulty of sanitizing the tree is not to be underestimated...

You are given a string (e.g. from a web input field or something) and need to be able to make fast calls to that string.

Keep in mind that if you fail to sanitize the tree properly, it would result in a very easy attack vector against your web server.

Thank you for the very thorough answer. – Julius Apr 10 '17 at 08:54 — Julius, Apr 10 '17 at 08:54

Andrzej Pronobis · Answer 2 · 2017-04-08T19:10:50.187

1

You can pre-compile the string used in the eval expression. Please note the code below is just to illustrate the concept, and the specific example can be realized by evaluating a lambda (as suggested in the comments).

fc = compile('b*np.exp(a*x)', '<string>', 'eval')
def f_faster(a, b, x):
    return eval(fc)

Then:

x = np.linspace(0, 1, 10)
print(timeit.timeit('f_faster(1,2,x)', "from __main__ import f_faster, x, fc", number=10000))
print(timeit.timeit('f_fast(1,2,x)', "from __main__ import f, x, f_fast", number=10000))

gives:

0.0241389274597
0.0203421115875

edited Apr 08 '17 at 19:10

answered Apr 08 '17 at 19:03

Andrzej Pronobis

33,828
17
76
92

Thank you! I had never used compile before. This solves my immediate problem. – Julius Apr 10 '17 at 08:54

score 1 · Answer 3 · answered Apr 08 '17 at 19:34

Another approach which might be faster depending on how many times you need to evaluate the expression vs converting it from a string into a function.

>>> from sympy import *
>>> from sympy.utilities.lambdify import lambdify
>>> f = lambdify((x,a,b), sympify('b*exp(a*x)'))
>>> f(1,1,1)
2.7182818284590451

sympify takes a string and returns an expression meaningful to sympy. Notice that this is still risky since, AFAIK, sympify uses eval. On the other hand, using sympy does give you access to a host of symbolic algebra processing facilities.

EDIT: Almost forgot: this code provides a function that uses the version of exp from the math library. To use the one from np is easy enough; please see the lambdify doc.

Python: fast functions from string evaluations?

3 Answers3