0

I've been trying unsuccessfully to implement __import__() and reload() for my program but I can't get it to work. I have a string that I write to a .py file (module) then I load that as a module. Then I make changes to that .py file (module) and write to it, but I can't seem to get the return values to change in that newly updated module. Here is my code:

str1 = '''
def run():
    return 10

'''
f = open('mycode.py','w')
f.write(str1)
f.close()
mymodule = __import__('mycode')

es = 'number = mymodule.run()'
exec(es)
print "number", number


str2 = '''
def run():
    return 99

'''
f = open('mycode.py','w')
f.write(str2)
f.close()

mymodule = reload(mymodule)
mymodule = __import__('mycode')

es = 'number = mymodule.run()'
exec(es)
print "number", number


OUTPUT:
>> number 10
>> number 10 # should be 99

I have looked here: Re-import the same python module with __import__() in a fail-safe way

here:

reloading module which has been imported to another module

and here:

How do I unload (reload) a Python module?

But I was not able to come up with a solution. Any help would be appreciated. Thanks.

Paul

EDIT

The reason I am using exec(es) is because I want to customize es if I have multiple parameters. Here's an example:

p = [2,1]
p2 = [3,4,5]
p3 = [100,200,300,500]

str1 = '''
def run(x,y):
        return x + y

'''

with open('mycode.py','w') as f:
    f.write(str)
import mycode as mymodule

# how do i do this dynamically, 
# how to handle it when # of paremeters for run() change?
print mymodule.run(p[0],p[1])  # hard-coded parameter passing
# or, print mymodule.run(p[0],p3[1],p[2]) # if run() is changed

So the issue is my run() can have different parameters. It can be run(x,y) or run(larry, moe, curly, hickory, dickory, dock). How do I dynamically pass multiple parameters to run()? Thanks.

Community
  • 1
  • 1
Paul
  • 2,409
  • 2
  • 26
  • 29
  • 1
    I know I'm not answering the question, but this is not the way to go if you want to create new behavior at runtime. In Python, you can redifine and patch classes on the run, without the need to `reload()`. The `reload` mechanism is flimsy at best -- it's meant for the REPL, not for programmatic usage. – salezica Mar 09 '13 at 00:12
  • If you really want to replace a module, you can always access `sys.modules`. This is what `python-sh` does, for example – salezica Mar 09 '13 at 00:13
  • Meanwhile, why are you using `exec(es)` here instead of just `number = mymodule.run()`? Are you expecting it to do something different? (If so, what?) Or just adding even more complexities to make the problem more difficult? – abarnert Mar 09 '13 at 00:13
  • I am using exec() because I want to be able to change the parameters of run() e.g. run(10,30). I didn't include that in my explanation. – Paul Mar 09 '13 at 00:16
  • You don't need `exec` to change the parameters in Python. You can pass around functions, create partials or lambdas that bind parameters to functions, etc. The fact that you're trying to do that is a sign that you're way off on the wrong tangent. As is the fact that you're using `__import__` instead of just `import`. (If you have a module name rather than a pathname, and you have it as a literal, you do not need `__import__`.) – abarnert Mar 09 '13 at 00:18
  • That could be one way. – Paul Mar 09 '13 at 00:22
  • _What_ could be one way? – abarnert Mar 09 '13 at 00:25
  • 1
    The way you handle it calling a function with a dynamic number of parameters is simple: do `mymodule.run(*p)`. That will do `mymodule.run(p[0], p[1])` if there are two elements, or `mymodule.run()` if there are no elements. No need for `exec`. – abarnert Mar 09 '13 at 01:30
  • It's becoming apparent that this is a classic [XY problem](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). You don't actually want to know how to re-create and reload a module; you just think that's the solution to your real problem, when it isn't. Tell us what your real problem is—what you're actually trying to do—with a specific example that shows why you think it's hard, and we can give you a specific solution that shows how it's very easy and doesn't require any of the stuff you're trying to do. – abarnert Mar 09 '13 at 01:49

2 Answers2

0

Your actual problem here is pretty simple. And I'll explain how to fix it. But really, you should not be doing this. Whatever you're trying to do, you're doing it wrong.

It's not that there's no reason anyone could ever possibly want to do such a thing. But I'm pretty confident that anyone who uses exec because he doesn't know you can pass functions around as first-class values, calls __import__ with a literal module name, etc. does not have such a reason. If you explain what you're actually trying to do, we can explain it.

But meanwhile:

When you do the first import, it's creating a mycode.pyc file, and the reload is just reloading that file.

You can verify this by removing mycode.pyc, then running python -B reloadtest.py instead of python reloadtest.py. And that obviously gives you one possible fix.

Alternatively, you can explicitly os.unlink('mycode.pyc') right before calling reload. I think most people would consider this horribly unpythonic, but I don't think it's any more so than what you're trying to do in the first place.

Finally, if you created a new mycode.py file instead of rewriting the existing file in place, I think that would solve the problem.


Meanwhile, your code is exactly equivalent to the following. (To address your confusion in the comments: Exactly identical means it will have the exact same problem, and the exact same solutions will solve it.) It's also a lot simpler, more pythonic, and easier to debug, because it only involves one huge antipattern (using reload anywhere but the interactive interpreter) instead of four:

str1 = '''
def run():
    return 10

'''
with open('mycode.py', 'w') as f:
    f.write(str1)
import mycode as mymodule

print "number", mymodule.run()

str2 = '''
def run():
    return 99

'''
with open('mycode.py', 'w') as f:
    f.write(str2)

reload(mymodule)

print "number", mymodule.run()

In your edit, you explain that you're using exec because you want to pass a dynamically-variable number of parameters:

It can be run(x,y) or run(larry, moe, curly, hickory, dickory, dock). How do I dynamically pass multiple parameters to run()?

Simple. Just put x, y or larry, moe, curly, hickory, dickory, dock into a tuple, list, or other sequence, and use run(*seq).

For example, I assume you have code something like this:

es = 'number = mymodule.run({})'.format(','.join(params))
exec(es)

You can replace that with:

number = mymodule.run(*params)

In fact, almost any case you can imagine where you think you need exec, there's a better way to do it. If you can't figure it out, you can do a quick search on Google or StackOverflow and, failing that, just ask how to do it. Using exec for dynamic code generation is idiomatic for tcl, and used to be for PHP and JavaScript years ago, but it's almost never the right thing to do in Python.

And, as you can probably guess by now, the same is true for dynamically creating modules to import/reload/execfile/etc.

For example, your real code is probably trying to do something like this:

def make_new_module(n):
    str = '''
    def run():
        return {}        
    '''.format(n)
    f = open('mycode.py','w')
    f.write(str1)
    f.close()
    return __import__('mycode')

But you can just define the function on the fly:

def make_new_function(n):
    def run():
        return n
    return run

If you really need it to be named mymodule.run, you can easily create a namespace like a class, class instance, namedtuple, etc. to stick it in.

You can even create a new module on the fly, if you really need to (although I'm willing to bet you don't need to):

import types
def make_new_module(n):
    def run():
        return n
    mymodule = types.ModuleType('mymodule')
    mymodule.run = run
    return mymodule

And rebinding a function in an existing module is even easier:

def change_mymodule_run(n):
    def run():
        return n
    mymodule.run = run

And yes, modules are first-class values, just like functions, so you can do it just as easily if module is a dynamic parameter instead of a static value:

def change_module_run(module, n):
    def run():
        return n
    module.run = run
abarnert
  • 354,177
  • 51
  • 601
  • 671
  • I appreciate the suggestions and corrections to more Pythonic code. I have updated my original question to explain why I am using exec(). – Paul Mar 09 '13 at 01:27
  • For some reason the above code still only gives me output: >> number 10 not 99. I am running Python 2.7.1. :( – Paul Mar 09 '13 at 01:42
  • @Paul: As I explained, the above code is identical to your original code. So of course it has the exact same problem. And it can be solved the exact same way: `python -B`, `os.unlink('mycode.pyc')`, etc. Or, better, stop trying to create modules dynamically in the first place. – abarnert Mar 09 '13 at 01:47
  • The reason I am trying to dynamically create modules is because I am embedding python code into xml files for a tax accounting quiz program I am making. I need to embed logic and variables into each unique question. – Paul Mar 09 '13 at 02:08
  • @Paul: Even for a case like that, you're usually better off embedding something other than Python source text. First, you can always `pickle` a function instead of dumping its text. But usually there's something even better. For example, if the functions only differ based on some data values, embed the values. If they need to be a bit more complex, write a simple domain-specific language. And even in the rare cases where you really do need to execute arbitrary Python source code dynamically, `exec` is usually a better fit than `import`. – abarnert Mar 10 '13 at 02:07
  • "Write a.. language." For me there is dimishing returns on how much more effort I need to put in this. I will stick to this non-ideal fix. I am the only one who has to live with it since I will be the only user. Thanks for the suggestions. – Paul Mar 10 '13 at 02:42
  • @Paul: Writing a domain-specific language is only one of many options; just because that one isn't the right one doesn't mean that none of them are. For example, storing pickled functions instead of source is a lot simpler than what you're trying to do, almost trivial in fact. – abarnert Mar 10 '13 at 03:15
0

As some of the commenters have said, you probably don't need to be writing modules in order to have dynamic code. Here's an alternative implementation that creates two different versions of a run function, which give different results:

def run_function_factory(n):
    def run_function():
        return n
    return run_function

run1 = run_function_factory(10)
print("run1 returns", run1()) # prints "run1 returns 10"

run2 = run_function_factory(99)
print("run2 returns", run2()) # prints "run2 returns 99"

Obviously you're actual code will be something more complicated, but a similar style of coding (treating functions as first-class object) should be possible for almost any variation you need. For instance, if run takes two arguments, a and b and does some arithmetic with them and n before returning, you'd use something like this:

def run_function_factory(n):
    def run_function(a, b):
        return a*n + b

run1 = run_function_factory(10)
for x in range(3):
    for y in range(2,5):
        print(run1(x,y)) # prints 2,3,4,12,13,14,22,23,24 (one per line)
Blckknght
  • 100,903
  • 11
  • 120
  • 169