Right way to turn an input string into a callable function and store it as variable?

Question

I was testing some code to answer Possible to turn an input string into a callable function object in Python? and the question got closed as duplicate but I could not find the answer to that question in the source answers.

There are many answers on SO similar to this but none address the issue how to return a function from string. exec() doesn't return a function. It just executes arbitrary code.

my question is, is this right approach to convert string to function and return as a function object?

my_string = "def add5(x):return x + 5"

def string_to_func(my_string):
    exec(my_string)
    for var_name, var in locals().items():
        if callable(var):
            return var
    print("no callable in string")
    #fallback code.

add5 = string_to_func(my_string)
print(add5(3))

P.S.: I will delete this question if the original question gets reopened.

This is sort of answer to the question in the link. So we will have to ask the original OP why don't he try using lambda, instead of def ? Thanks any way. I hate lambdas. I am too dumb to understand it. — Rahul, Jan 31 '19 at 05:13
when you define a function and store it in a string, there is no means by which you will be able to use the name provided to the function once, it gets converted to a string. Hence, you can make this approach less confusing and more reader-friendly, by the use of lambdas. something like this: `my_string = "lambda x: x + 5"`. — taurus05, Jan 31 '19 at 05:17
Trivially `def wrapper(string): def wrapped(): return string; return wrapped` — tripleee, Jan 31 '19 at 05:23
Possible duplicate of [Python: dynamically create function at runtime](https://stackoverflow.com/questions/11291242/python-dynamically-create-function-at-runtime) — metatoaster, Jan 31 '19 at 05:23
If you update as per @YassineAddi you code works fine and outputs the `no callable in string" and "8". — DaveStSomeWhere, Jan 31 '19 at 05:27
@metatoaster: It's different. you have to carefully read the entire question and the answer on the source question. you can't just google and mark it as the duplicate. — Rahul, Jan 31 '19 at 05:45
@Rahul did you see the various answers and comments there, what you are doing is effectively dynamically construct some anonymous function as outlined by [this thread](https://stackoverflow.com/questions/10303248/true-dynamic-and-anonymous-functions-possible-in-python), which was also linked as a possible duplicate to the one I marked as a duplicate. Perhaps this is a better duplicate candidate to the higher level one which I linked. — metatoaster, Jan 31 '19 at 05:55
In all the case you will have to know what is the name of the function you are executing from a string. None answeres the OP question how one return function from a string as a variable. Please point me to the answer which does that. — Rahul, Jan 31 '19 at 06:00
would some string processing that turns `my_string` into `add5 = lambda x...` work? — sam46, Jan 31 '19 at 06:07
@Rahul Did you see the examples using [`types.FunctionType`](https://docs.python.org/3/library/types.html#types.FunctionType)? You have to call `compile` on the string based on some template, and then extract the `__code__` attribute from the known name that you assigned, and then create a new `FunctionType` object with the appropriate arguments and then assign/return it as a value. — metatoaster, Jan 31 '19 at 06:25
Admittedly, if you are looking for a Python 3 specific answer that would not be a sufficient duplicate candidate. — metatoaster, Jan 31 '19 at 06:27
I have never used python2. I will check your guidance. May be if what you are saying is better way please answer it. — Rahul, Jan 31 '19 at 06:28
I am saying a better way. I am not experienced enough to judge the right way — Rahul, Jan 31 '19 at 07:59

metatoaster · Accepted Answer · 2019-01-31T11:02:38.310

Roughly speaking, if you assume that inputs are fully trusted, and where the provided string is of valid Python syntax that will produce a single callable function, and if exec cannot be used, something like this can be provided

import ast
import types

def string_to_function(source):
    tree = ast.parse(source)
    if len(tree.body) != 1 or not isinstance(tree.body[0], ast.FunctionDef):
        raise ValueError('provided code fragment is not a single function')
    co = compile(tree, 'custom.py', 'exec')
    # first constant should be the code object for the function
    return types.FunctionType(co.co_consts[0], {})

Example:

f = string_to_function("""
def my_function(x):
    return x + 5
"""
)

print('f = %d' % f(5))

Output

f = 10

The code ensures that the provided source is of a single function definition, and makes assumption of the organisation of the generated bytecode (i.e. only works for the compiler built into the current versions of Python, where the generated bytecode places the code object for the single function that was defined in the source in the 0th element in co_consts). The previous version to this answer made use of exec (which questioner "is not a big fan of exec anyway"), done in a way that binds the results into a specific target and this method should be more reliable as it does not touch this lower level structure, though the sanity checks using ast module used here could be included instead with that original version.

Also note that this answer is only applicable for Python 3+.

Further Edit:

I am actually still a little miffed by the remark on the usage of exec being assumed to execute arbitrary code (on fixed inputs) simply because what it actually does is often very misunderstood. In this particular case, if it is verified that only specific source is accepted, it doesn't necessarily mean every statement is executed immediately. This is especially true for this case (which isn't properly guaranteed in my original lazy answer, but this is where actual understanding of what the framework is actually doing is important for doing anything that involves dynamic compilation of code within the language framework, and given that more level of "safety" is desired (executing function immediately after the fact negates it hence I didn't implemented originally) using ast is done in the edit and it is now objectively better).

So what exactly does calling exec(co) do like essentially what the original example did when given the source input of an single function definition? It can be determined by looking at the bytecode like so:

>>> dis.dis(co)
  2           0 LOAD_CONST               0 (<code object my_function at 0x7fdbec44c420, file "custom.py", line 2>)
              2 LOAD_CONST               1 ('my_function')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (my_function)
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE

All it does is to load the code object, make it into a proper function and assign the result to my_function (on the currently relevant scope), and essentially that's it. So yes, the correct way is to verify that the source is definitely a function definition like so here (as verification through checking the AST is more safe than a more naive verification that only one statement is present), then running exec on a specific dict and extract the assignment from there. Using exec this way is not inherently less (or more) safe in this instance, given that any function that was provided would be executed immediately anyway.

Which OP are you referring to, your code or the ones I linked in some other thread? You see this is why I had been saying the essence of your question had been answered elsewhere. — metatoaster, Jan 31 '19 at 07:08
Try to use it on a variety of inputs and come to your own conclusion, you asked the question... — metatoaster, Jan 31 '19 at 07:09
OK. I thought you already know it. I am not a big fan of `exec` anyway. Thanks for the sharing your experience. — Rahul, Jan 31 '19 at 07:11
@Rahul at the very least, splitting out the process like how I've done it precisely identify the local names that were defined, thus further processing may be done so in that sense it does more, but whether it makes it "better" completely depends on what you intend to use this for. — metatoaster, Jan 31 '19 at 07:11
Alternatively, you can manually verify/build everything with the help of the [`ast`](https://docs.python.org/3/library/ast.html) module. You also did not specify that you do not wish to use `exec` in the code you wish to write. — metatoaster, Jan 31 '19 at 07:13

Right way to turn an input string into a callable function and store it as variable?

1 Answers1

Linked