How to reload bytecode from the datastore in app engine

Question

I created an import hook (based on PEP 302), to dynamicly load python bytecode (compiled python code) from the database.

import dyn_imports.test   #@UnresolvedImport

This works fine, but if I change the module and reload the module, the module reloads but I do not see the changes.

This is how I create the bytecode :

code_object = compile(db.Text(dyn.py_source).encode(), dyn.name.encode(), 'exec')
key_name = 'dyn_imports.' + name
dynmod = models.Dynmods(key_name = key_name, name = key_name, bytecode = db.Blob(marshal.dumps(code_object)))
dynmod.put()
reload(sys.modules[key_name])             # reload the updated code

The reload() executes the load_module part of the import hook.
This is the simplified version of the load_module function in the import hook class:

def load_module(self, mod_name):
    mod = imp.new_module(mod_name)
    mod.__loader__ = self
    dynmod = models.Dynmods.get_by_key_name(mod_name)    # load from DB
    mod.__file__ = mod_name
    exec marshal.loads(db.Blob(dynmod.bytecode)) in mod.__dict__
    sys.modules[mod_name] = mod
    return mod

When I shutdown the instance(s), it works fine.

Update : I also have tested the import loader using py source instead of bytecode. This did not work either. I have done most of the testing on the dev server.

I have created a Jinja2 module loader before, which loads Jinja compiled templates (= py source) from the datastore. When I re-create the Jinja environment and delete the modules (compiled templates) from sys.modules, the new modules are reloaded, without instance restart. So what am I doing wrong here?

voscausa · Accepted Answer · 2012-12-01T02:40:20.220

I have found a solution. Besides reloading the changed module, I have to reload all modules that use (imported) this module. I have done some tests and it works fine. The correct order is also important. So now I have to figure out, how to create an import dependency graph and use it to reload modules.

But that is a new question.

I create a list of dependencies, when modules import the dynamic module. Using this list, I can reload the dependent modules. I also have to flag the other instances of my GAE appid. These other instances have to reload the flagged dynamic module (if used) and reload the dependent modules for the dynamic module in this other instance.

To create unique instance ids for every instance, I added the class variable :

instance = datetime.datetime.utcnow()

to the import hook class. Memcache is used by the setter instance to flag the other instances. This flag contains per reloaded dynamic module: the flag setter instance and the reload datetime stamp of the dynamic module. Other (flag getter) instances can check, if they have to reload a dynamic module.

How to reload bytecode from the datastore in app engine

1 Answers1