How to speed up a repeated execution of exec()?

Question

I have the following code:

for k in pool:
    x = []
    y = []

    try:
        exec(pool[k])
    except Exception as e:
        ...
    do_something(x)
    do_something_else(y)

where pool[k] is python code that will eventually append items to x and y (that's why I am using exec instead of eval).

I have tried already to execute the same code with pypy but for this particular block I don't get much better, that line with exec is still my bottleneck.

That said, my question is: Is there a faster alternative to exec?

If not, do you have any workaround to get some speed up in such a case?

--UPDATE--

To clarify, pool contains around one million keys, to each key it is associated a script (around 50 line of code). The inputs for the scripts are defined before the for loop and the outputs generated by a script are stored in x and y. So, each script has a line in the code stating x.append(something) and y.append(something). The rest of the program will evaluate the results and score each script. Therefore, I need to loop over each script, execute it and process the results. The scripts are originally stored in different text files. pool is a dictionary obtained by parsing these files.

P.S. Using the pre-compiled version of the code:

for k in pool.keys():
    pool[k] = compile(pool[k], '<string>', 'exec')

I have got 5x speed increase, not much but it is something already. I am experimenting with other solutions...

I assume you know what you're doing and are fully aware of the potential [security risks of using `exec`](http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html)... You can get some speedup if you [`compile`](https://docs.python.org/3/library/functions.html#compile) the code you pass to `exec`, especially if you are executing the same code multiple times. See [this answer](http://stackoverflow.com/a/29456463/4014959) for a whole lot of info about `eval`, `exec`, and `compile`. — PM 2Ring, Jul 09 '16 at 17:07
I am aware of the security risks, I am just running code locally and is just a bunch of math formulas. I will check `compile`. Thanks. — alec_djinn, Jul 09 '16 at 17:13
If `exec()` really is the bottleneck, avoid using it. Make `pool` a dictionary of functions that each contain the code you want executed, and then just call the proper one with `pool[k](x, y)`. — martineau, Jul 09 '16 at 17:30
The code does contains also if/else and other stuff, but more important, I get the code from a textfile, so I have it as string in the first place. — alec_djinn, Jul 09 '16 at 17:32
Are you re-executing each item from pool, or are they just throw-away? — Antti Haapala -- Слава Україні, Jul 09 '16 at 17:42
re-executing, indeed precompilation may work.. I am testing it now. — alec_djinn, Jul 09 '16 at 17:46
I didn't understand why not write a script that would iterate on the the text files and generate a python file with functions and then import that file. — AturSams, Jun 21 '22 at 11:18
@AturSams because of IO limitations. Writing millions of small files on the HD it isn't a good colution. — alec_djinn, Jun 23 '22 at 08:34

Dalen · Accepted Answer · 2020-06-26T12:50:10.073

3

If you really need to execute some code in such manner, use compile() to prepare it.

I.e. do not pass raw Python code into exec but compiled object. Use compile() on your codes before to make them Python byte compiled objects.

But still, it'll make more sense to write a function that will perform what you need on an input argument i.e. pool[k] and return results that are corresponding to x and y.

If you are getting your code out of a file, you have also IO slowdowns to cope with. So it would be nice to have these files already compiled to *.pyc.

You may think about using execfile() in Python2.

An idea for using functions in a pool:

template = """\
def newfunc ():
%s
    return result
"""

pool = [] # For iterating it will be faster if it is a list (just a bit)
# This compiles code as a real function and adds a pointer to a pool
def AddFunc (code):
    code = "\n".join(["    "+x for x in code.splitlines()])
    exec template % code
    pool.append(newfunc)

# Usage:
AddFunc("""\
a = 8.34**0.5
b = 8
c = 13
result = []
for x in range(10):
    result.append(math.sin(a*b+c)/math.pi+x)""")

for f in pool:
    x = f()

edited Jun 26 '20 at 12:50

answered Jul 09 '16 at 17:11

Dalen

4,128
1
17
35

x and y are lists, they store the some value that will be generated by the exec(code), so part of the code in it is x.append(something). I will try `exec(compile(code))` – alec_djinn Jul 09 '16 at 17:16
Woops, sorry, saw wrong. Trouble with encoding. I thought there were ( ) instead of [ ]. Why aren't you x.append()-ing in the loop? – Dalen Jul 09 '16 at 17:19
@alec_djinn : There, i edited the A to correct the mistake. Sorry again. – Dalen Jul 09 '16 at 17:22
yes, I know that x and y are initialized to [] at every iteration, it's ok it's meant to do like that. Basically, pool contains lots of different math functions that are supposed to produce some results, the results are appended to x and y and then other code (`...`) will evaluate how good the results are etc etc... At every iteration a different set of functions are evaluated. – alec_djinn Jul 09 '16 at 17:22
Then I may recommend writing these as functions and saving pointers to these functions into pool. Then you can do pool["one_func"]() etc. etc. How do you generate these functions in advance is not important. You may still use exec but your functions will be global and only pointers passed through a loop. Very fast. – Dalen Jul 09 '16 at 17:31
@alec_djinn : I wrote a quick example that implicitly avoids compile(), but in the end it comes to same really. See my edit. This requires your code to have specific var named result always there. :D If you are loading from files, well, security... You said you know the risks, so no more comments on that. :D – Dalen Jul 09 '16 at 17:59
I'll try your example.. looks a bit cumbersome to be honest but if it does the job I am happy with that. – alec_djinn Jul 09 '16 at 18:06
Yep, because it borders on meta programming. You may pull same trick in more than one way, so feel free to experiment until you are happy with code look and speed. If something is unclear, just ask. Good luck! – Dalen Jul 09 '16 at 21:56

How to speed up a repeated execution of exec()?

1 Answers1