Use user provided python code during runtime

Question

I'm developing a system that operates on (arbitrary) data from databases. The data may need some preprocessing before the system can work with it. To allow the user the specify possibly complex rules I though of giving the user the possibility to input Python code which is used to do this task. The system is pure Python.

My plan is to introduce the tables and columns as variables and let the user to anything Python can do (including access to the standard libs). Now to my problem:

How do I take a string (the user entered), compile it to Python (after adding code to provide the input data) and get the output. I think the easiest way would be to use the user-entered data a the body of a method and take the return value of that function a my new data.

Is this possible? If yes, how? It's unimportant that the user may enter malicious code since the worst thing that could happen is, that he screws up his own system, which is thankfully not my problem ;)

Adam Batkin · Accepted Answer · 2011-08-02T18:54:12.940

4

Python provides an exec() statement which should do what you want. You will want to pass in the variables that you want available as the second and/or third arguments to the function (globals and locals respectively) as those control the environment that the exec is run in.

For example:

env = {'somevar': 'somevalue'}
exec(code, env)

Alternatively, execfile() can be used in a similar way, if the code that you want executed is stored in its own file.

If you only have a single expression that you want to execute, you can also use eval.

edited Aug 02 '11 at 18:54

answered Aug 02 '11 at 18:28

Adam Batkin

51,711
9
123
115

1

`eval()` does __not__ do what he wants; `exec` does. `eval()` is _only for expressions_, `exec` is for arbitrary code. – agf Aug 02 '11 at 18:39

score 2 · Answer 2 · answered Aug 02 '11 at 18:40

Is this possible?

If it doesn't involve time travel, anti-gravity or perpetual motion the answer to this question is always "YES". You don't need to ask that.

The right way to proceed is as follows.

You build a framework with some handy libraries and packages.

You build a few sample applications that implement this requirement: "The data may need some preprocessing before the system can work with it."

You write documentation about how that application imports and uses modules from your framework.

You turn the framework, the sample applications and the documentation over to users to let them build these applications.

Don't waste time on "take a string (the user entered), compile it to Python (after adding code to provide the input data) and get the output".

The user should write applications like this.

from your_framework import the_file_loop
def their_function( one_line_as_dict ):
    one_line_as_dict['field']= some stuff
the_file_loop( their_function )

That can actually be the entire program.

You'll have to write the_file_loop, which will look something like this.

def the_file_loop( some_function ):
    with open('input') as source:
        with open('output') as target:
            for some_line in source:
                the_data = make_a_dictionary( some_line )
                some_function( the_data )
                target.write( make_a_line( the_data ) )

By creating a framework, and allowing users to write their own programs, you'll be a lot happier with the results. Less magic.

maaudet · Answer 3 · 2011-08-02T18:59:09.387

1

2 choices:

You take his input and put it in a file, then you execute it.
You use exec()

edited Aug 02 '11 at 18:59

answered Aug 02 '11 at 18:28

maaudet

2,338
4
20
28

score 0 · Answer 4 · answered Aug 02 '11 at 18:32

If you just want to set some local values and then provide a python shell, check out the code module.

You can start an instance of a shell that is similar to the python shell, as well as initialize it with whatever local variables you want. This would assume that whatever functionality you want to use the resulting values is built into the classes you are passing in as locals.

Example:

shell = code.InteractiveConsole({'foo': myVar1, 'bar': myVar2})

score 0 · Answer 5 · answered Aug 02 '11 at 18:37

What you actually want is exec, since eval is limited to taking an expression and returning a value. With exec, you can have code blocks (statements) and work on arbitrarily complex data, passed in as the globals and locals of the code.

The result is then returned by the code via some convention (like binding it to result).

score -1 · Answer 6 · answered Aug 02 '11 at 18:34

well, you're describing compile()

But... I think I'd still implement this using regular python source files. Add a special location to the path, say '~/.myapp/plugins', and just __import__ everything there. Probably you'll want to provide some convenient base classes that expose the interface you're trying to offer, so that your users can inherit from them.

Use user provided python code during runtime

6 Answers6