5

Suppose I need to create my own small DSL that would use Python to describe a certain data structure. E.g. I'd like to be able to write something like

f(x) = some_stuff(a,b,c)

and have Python, instead of complaining about undeclared identifiers or attempting to invoke the function some_stuff, convert it to a literal expression for my further convenience.

It is possible to get a reasonable approximation to this by creating a class with properly redefined __getattr__ and __setattr__ methods and use it as follows:

e = Expression()
e.f[e.x] = e.some_stuff(e.a, e.b, e.c)

It would be cool though, if it were possible to get rid of the annoying "e." prefixes and maybe even avoid the use of []. So I was wondering, is it possible to somehow temporarily "redefine" global name lookups and assignments? On a related note, maybe there are good packages for easily achieving such "quoting" functionality for Python expressions?

KT.
  • 10,815
  • 4
  • 47
  • 71
  • "annoying "e." prefixes"? They seem absolutely essential to understanding what's going on. If `some_stuff` is not a function, but a secret method of an anonymous object, I'm stumped as to how anyone can learn to use this. – S.Lott Apr 17 '10 at 01:19
  • Well, learning to use this is an unrelated matter of appropriate documentation. Note that in my example some_stuff is __not__ refering to an existing function of the "expression" object either - it will be "created dynamically" with the help of `__getattr__`. What I am essentially seeking for is a way of creating sexy syntax sugar that could replace stuff like, say `add_new_formula(['f','x'],['some_stuff','a','b','c'])` by something more readable. One of the answers mentions Sage, see how it uses altered Python syntax for a live example. – KT. Apr 17 '10 at 03:32
  • 1
    @KT: Your various examples are all over the map. One assumes too much, one may have way too many needless object references to the same object, and the other has way too many quotes and brackets. Is there anything you can provide that would show a working example of what you're trying to do with the standard object classes? Could you omit the hyperbole? Could you provide working classes so we can help streamline the syntax? – S.Lott Apr 17 '10 at 11:40
  • Ian has provided an exhaustive answer - I was essentially looking for the `exec .. in ..` expression - so there's no question to worry about any more. If you want to better understand the motivation, then let me add that I was simply seeking for a fun way to make Python "parse" a certain configuration file, which used essentially Python syntax, but none of the identifiers or assignments corresponded to "real" Python objects or assignments (e.g. some of them have to be proxied to an external system). I think I got my answer. Thanks for trying to help, though, still! – KT. Apr 17 '10 at 13:08
  • 1
    @KT: If you're not going to clarify the question, does that mean you don't think ordinary folks should understand your question? I find it troubling when people don't want their question to be clear and complete. That seems to devalue SO for other folks who are trying to tackle the same problem. – S.Lott Apr 19 '10 at 20:03
  • I believe it's clear enough, especially considering the remarks added here and the availability of the accepted answer. I don't see a way to be more clear and I'm sure anyone tackling the same problem will find this answer valuable. Sorry. – KT. Apr 20 '10 at 05:45

3 Answers3

3

I'm not sure it's a good idea, but I thought I'd give it a try. To summarize:

class PermissiveDict(dict):
    default = None

    def __getitem__(self, item):
        try:
            return dict.__getitem__(self, item)
        except KeyError:
            return self.default

def exec_with_default(code, default=None):
    ns = PermissiveDict()
    ns.default = default
    exec code in ns
    return ns
Ian Bicking
  • 9,762
  • 6
  • 33
  • 32
  • 1
    Awesome! "Exec ... in ..." is more or less what I was looking for. I can now put this stuff into an annotation and have some magic happening inside a given function. It's probably impossible to make it more syntax-sugary than that. Thanks! – KT. Apr 17 '10 at 03:43
  • Heh, unfortunately this solution works differently depending on where you get the code object from. If it is a string or the result of a compile() things work as expected. However, if the code object is something like my_function.func_code, this fails due to the peculiarities of variable binding within a function - the local variables won't be looked up in the provided environment. It can be overcome by bytecode mangling (see code_all_variables_dynamic here: http://code.activestate.com/recipes/498242/), though. This makes the whole thing way more sophisticated than it could have been. – KT. Apr 17 '10 at 09:04
  • Yes... these problems won't ever really stop, which is why it would be better to work inside the scope of what's available without hacks if at all possible. – Ian Bicking Apr 17 '10 at 18:44
2

You might want to take a look at the ast or parser modules included with Python to parse, access and transform the abstract syntax tree (or parse tree, respectively) of the input code. As far as I know, the Sage mathematical system, written in Python, has a similar sort of precompiler.

AKX
  • 152,115
  • 15
  • 115
  • 172
  • Thanks, that's one possibility indeed. Nonetheless it's not exactly what I'd like since it would require me to separate my "expressions to be parsed" from the usual python by passing them around as strings, which would not be as awesome if it'd be possible to keep using the standard parsing and simply redefine evaluation behaviour (`__getattr__` and the like). – KT. Apr 17 '10 at 00:28
-1

In response to Wai's comment, here's one fun solution that I've found. First of all, to explain once more what it does, suppose that you have the following code:

definitions = Structure()
definitions.add_definition('f[x]', 'x*2')
definitions.add_definition('f[z]', 'some_function(z)')
definitions.add_definition('g.i', 'some_object[i].method(param=value)')

where adding definitions implies parsing the left hand sides and the right hand sides and doing other ugly stuff. Now one (not necessarily good, but certainly fun) approach here would allow to write the above code as follows:

@my_dsl
def definitions():
    f[x] = x*2
    f[z] = some_function(z)
    g.i  = some_object[i].method(param=value)

and have Python do most of the parsing under the hood. The idea is based on the simple exec <code> in <environment> statement, mentioned by Ian, with one hackish addition. Namely, the bytecode of the function must be slightly tweaked and all local variable access operations (LOAD_FAST) switched to variable access from the environment (LOAD_NAME).

It is easier shown than explained: http://fouryears.eu/wp-content/uploads/pydsl/

There are various tricks you may want to do to make it practical. For example, in the code presented at the link above you can't use builtin functions and language constructions like for loops and if statements within a @my_dsl function. You can make those work, however, by adding more behaviour to the Env class.

Update. Here is a slightly more verbose explanation of the same thing.

KT.
  • 10,815
  • 4
  • 47
  • 71