Preprocessing function text in runtime bofore compilation

Question

I decided to try to preprocess function text before it's compilation into byte-code and following execution. This is merely for training. I hardly imagine situations where it'll be a satisfactory solution to be used. I have faced one problem which I wanted to solve in this way, but eventually a better way was found. So this is just for training and to learn something new, not for real usage.

Assume we have a function, which source code we want to be modified quite a bit before compilation:

def f():
    1;a()
    print('Some statements 1')
    1;a()
    print('Some statements 2')

Let, for example, mark some lines of it with 1;, for them to be sometimes commented and sometimes not. I just take it for example, modifications of the function may be different.

To comment these lines I made a decorator. The whole code it bellow:

from __future__ import print_function


def a():
    print('a()')


def comment_1(s):
    lines = s.split('\n')
    return '\n'.join(line.replace(';','#;',1) if line.strip().startswith('1;') else line for line in lines)


def remove_1(f):    
    import inspect
    source = inspect.getsource(f)    
    new_source = comment_1(source)
    with open('temp.py','w') as file:
        file.write(new_source)
    from temp import f as f_new
    return f_new


def f():
    1;a()
    print('Some statements 1')
    1;a()
    print('Some statements 2')


f = remove_1(f) #If decorator @remove is used above f(), inspect.getsource includes @remove inside the code.

f()

I used inspect.getsourcelines to retrieve function f code. Then I made some text-processing (in this case commenting lines starting with 1;). After that I saved it to temp.py module, which is then imported. And then a function f is decorated in the main module.

The output, when decorator is applied, is this:

Some statements 1
Some statements 2

when NOT applied is this:

a()
Some statements 1
a()
Some statements 2

What I don't like is that I have to use hard drive to load compiled function. Can it be done without writing it to temporary module temp.py and importing from it?

The second question is about placing decorator above f: @replace. When I do this, inspect.getsourcelines returns f text with this decorator. I could manually be deleted from f's text. but that would be quite dangerous, as there may be more than one decorator applied. So I resorted to the old-style decoration syntax f = remove_1(f), which does the job. But still, is it possible to allow normal decoration technique with @replace?

You should be modifying the AST, not the source. – Ignacio Vazquez-Abrams Sep 02 '12 at 18:26 — Ignacio Vazquez-Abrams, Sep 02 '12 at 18:26

user4815162342 · Accepted Answer · 2012-09-02T19:04:35.293

1

One can avoid creating a temporary file by invoking the exec statement on the source. (You can also explicitly call compile prior to exec if you want additional control over compilation, but exec will do the compilation for you, so it's not necessary.) Correctly calling exec has the additional benefit that the function will work correctly if it accesses global variables from the namespace of its module.

The problem described in the second question can be resolved by temporarily blocking the decorator while it is running. That way the decorator remains, along all the other ones, but is a no-op.

Here is the updated source.

from __future__ import print_function

import sys


def a():
    print('a()')


def comment_1(s):
    lines = s.split('\n')
    return '\n'.join(line.replace(';','#;',1) if line.strip().startswith('1;') else line for line in lines)

_blocked = False

def remove_1(f):
    global _blocked
    if _blocked:
        return f
    import inspect
    source = inspect.getsource(f)    
    new_source = comment_1(source)
    env = sys.modules[f.__module__].__dict__
    _blocked = True
    try:
        exec new_source in env
    finally:
        _blocked = False
    return env[f.__name__]


@remove_1
def f():
    1;a()
    print('Some statements 1')
    1;a()
    print('Some statements 2')


f()

def remove_1(f):    
    import inspect
    source = inspect.getsource(f)    
    new_source = comment_1(source)
    env = sys.modules[f.__module__].__dict__.copy()
    exec new_source in env
    return env[f.__name__]

edited Sep 02 '12 at 19:04

answered Sep 02 '12 at 18:41

user4815162342

141,790
18
296
355

Why do we need to make a copy of the dictionary of modules dictionary? – ovgolovin Sep 02 '12 at 18:57
We don't have to, but it seemed like a good idea not to accidentally destroy something in the dictionary. I'll retract it in the next edit that also answers the 2nd question. – user4815162342 Sep 02 '12 at 19:00
Whoa! It works! If we don't use `copy`, `remove_1` may be turned into ordinary function without decoration. http://ideone.com/avF3P – ovgolovin Sep 02 '12 at 19:02
Oh, right, the reason I used `copy` was to avoid `exec` immediately destroying the original function, thereby making the decoration kind of redundant. In your case it doesn't matter, but it seems somewhat destructive. – user4815162342 Sep 02 '12 at 19:08
This is great. Thank you! Do I understand it right, that this is a sheer magic, and I shouldn't resort to it ever? – ovgolovin Sep 02 '12 at 19:15
I have one more question. Why `inspect.getsource` includes `@remove_1`? – ovgolovin Sep 02 '12 at 19:15
You are correct, you should never resort to this, there are so many ways it can go wrong. First, Python is not a language designed for source-level transformations, and it's hard to write it a transformer such as comment_1 without gratuitously breaking valid code. Second, this hack would break in all kinds of circumstances - for example, when defining methods, when defining nested functions, when used in Cython, when inspect.getsource fails for whatever reason. Python is dynamic enough that you really don't need this kind of hack to customize its behavior. – user4815162342 Sep 02 '12 at 19:30

score 0 · Answer 2 · edited May 23 '17 at 11:56

I'll leave a modified version of the solution given in the answer by user4815162342. It uses ast module to delete some parts of f, as was suggested in the comment to the question. To make it I majorly relied on the information in this article.

This implementation deletes all occurrences of a as standalone expression.

from __future__ import print_function
import sys
import ast
import inspect


def a():
    print('a() is called')


_blocked = False

def remove_1(f):
    global _blocked
    if _blocked:
        return f
    import inspect
    source = inspect.getsource(f)

    a = ast.parse(source) #get ast tree of f

    class Transformer(ast.NodeTransformer):
        '''Will delete all expressions containing 'a' functions at the top level'''
        def visit_Expr(self, node): #visit all expressions
            try:
                if node.value.func.id == 'a': #if expression consists of function with name a
                    return None #delete it
            except(ValueError):
                pass
            return node #return node unchanged
    transformer = Transformer()
    a_new = transformer.visit(a)
    f_new_compiled = compile(a_new,'<string>','exec')

    env = sys.modules[f.__module__].__dict__
    _blocked = True
    try:
        exec(f_new_compiled,env)
    finally:
        _blocked = False
    return env[f.__name__]


@remove_1
def f():
    a();a()
    print('Some statements 1')
    a()
    print('Some statements 2')


f()

The output is:

Some statements 1
Some statements 2

Preprocessing function text in runtime bofore compilation

2 Answers2

Linked