4

How can I run a multi-line string of code from inside python, additionally the string is generated at runtime and stored in either a string or an array.

Background

  • I have a function which dynamically generates random code snippets in high volume.
  • I wish to pass this code to a variable/string and evaluate syntax/return code.
  • Code can not be imported from file/api request etc as focus is on high throughput and speed.

Example

# my function returns random code snippets
myCode = myRandomCodeGenerator()

# I wish to perform some in fight validation (ideally RC or syntax)
someTestingFunction(myCode)

My attempts so far

I have seen solutions such as the one below, however since my code is generated dynamically I have issues formatting. I have tried generating \n and \r or adding """ in the string to compensate with no luck. 

code = """
def multiply(x,y):
    return x*y

print('Multiply of 2 and 3 is: ',multiply(2,3))
"""

exec(code)

I have also tried using the call function on a string yields similar formatting issues.

So far the best I can do is to perform a line by line syntax check and feed it a line at a time as shown below. 

import codeop
def is_valid_code(line):
    try:
        codeop.compile_command(line)
    except SyntaxError:
        return False
    else:
        return True

I suspect there may be some syntax tricks I could apply to preserve formatting of indentation and returns. I am also aware of the risk of generating dynamic code in runtime, and have a filter of allowed terms in the function.

Murchie85
  • 815
  • 2
  • 12
  • 27
  • how do you expect it to do anything if you cant get your spacing formatted correctly ... ? spacing is very important to python... i think this is an XY problem... where you are doing X to solve Y, but there is a much better path to Y... also whatever safeties you think you have in place... if this is user input data you are gonna get owned if someone really wants to – Joran Beasley May 23 '20 at 22:22
  • 1
    I agree, is there a way to preserve spacing/indentation when generated dynamically? The exec example i listed works fine when input manually. – Murchie85 May 23 '20 at 22:25
  • it(`exec`) does not modify the whitespace ... so generate code that has valid spacing – Joran Beasley May 23 '20 at 22:26
  • I already have, call and exec functions does not seem to recognise it. I validated this manually when passed to a string, is there a dynamic way to preserve multiple lines in a string, as in the triple quote method. – Murchie85 May 23 '20 at 22:29
  • your example above is a triple quoted string... and it works ... I guess i don't understand what you are asking – Joran Beasley May 23 '20 at 22:31
  • 1
    `exec("def multiply(x,y):\n return x*y\nprint('Multiply of 2 and 3 is: ',multiply(2,3))\n")` is the equivelent of the triple quoted string above, without triple quotes – Joran Beasley May 23 '20 at 22:34
  • 1
    Thank you Joran, I see where my code was failing; an automatic space was added after every \n as a safety margin. Even in the example above it fails if the print statement looks like this `return x*y\n print('Multiply of 2 and 3 is:` Much appreciated and I see both the reason and that this is now possible - thanks again. – Murchie85 May 23 '20 at 22:42
  • 1
    I did not see the obvious uin the above comments - or in the answer - but you are aware that using the `\n` special sequence in a code string will generate a newline character, don't you? Adding a `\n` to the end of a line is all that is needed for a multiline script. – jsbueno May 23 '20 at 22:50
  • Thanks Jsbueno, yes I think my issue was I wasn't sure it was even possible, and I was deliberately adding a space after the `\n` such as `\n print(..)` to avoid it being read as a single unit. I didn't realise python accounted for this. As such it was giving me errors anytime when defining a function. – Murchie85 May 23 '20 at 23:05

3 Answers3

1

Another way instead of eval and exec is to compile the code before and then to exec it:

Example:

import contextlib,sys
from io import StringIO 
@contextlib.contextmanager
def stdoutIO(stdout=None):
    old = sys.stdout
    if stdout is None:
        stdout = StringIO()
    sys.stdout = stdout
    yield stdout
    sys.stdout = old


def run_code(override_kale_blocks):
    compiled123 = []
    for b123 in override_kale_blocks:
        compiled123.append(compile(b123,"<string>","exec"))
    
    with stdoutIO() as s:
        for c123 in compiled123:
            exec(c123)
    return s.getvalue()
    


block0='''
import time
a=5
b=6
b=a+b
'''
block1='''
b=a+b
'''
block2="print(b)"
blocksleep='''
print('startsleep')
time.sleep(1)
print('donesleep')
'''
pc  = (block0,blocksleep,block1,block2)
cb = []
print('before')
output= run_code(pc)
print(output)
print('after')
print("Hello World!\n")

source: https://stackoverflow.com/a/3906390/1211174

oak
  • 2,898
  • 2
  • 32
  • 65
0

Question was answered by Joran Beasley.

Essentially, my issue was I was adding a space after \n as other compilers get confused, python however takes this into account. Thus if you add a space after a return carriage in a def statement, it will produce syntax error.

I have kept this incase others encounter similar issues.

Murchie85
  • 815
  • 2
  • 12
  • 27
-1

Try using '\t' instead of placing tabs and spaces for the indentation, that may preserve your formatting.

code = """
def multiply(x,y):
\t return x*y

print('Multiply of 2 and 3 is: ',multiply(2,3))
"""
exec(code)
Chems Eddine
  • 153
  • 6
  • 1
    `\t` simply expands to a tab character - even in generated code it should be avoided, as there are several pitfalls. – jsbueno May 23 '20 at 22:48