IronPython - Is there a way to load a python script from the database or embedded from a string resource?

Question

Is there a way to catch some kind of even from IronPython when it attempts to resolve/load a module that it cannot find?

We want to store modules in a database or embed them in a DLL and import them by calling import mymodule. We don't want to involve the filesystem at all. When we say import something, we do not want it looking in \Lib or any filesystem.

The following code works fine for loading a chunk of Python embedded in a DLL. It works only when there are no imports.

    var myScript = new StreamReader(Assembly.GetAssembly(this.GetType()).GetManifestResourceStream("Resource.Name.To.My.Script")).ReadToEnd()
    var engine = IronPython.Hosting.Python.CreateEngine();
    var scope = engine.CreateScope();
    var result = engine.Execute(myScript, scope);

And the python code:

print('hello world')

If there is an import, things do not work. It doesn't know how to resolve the location of the module.

The only way I can make it work is by making sure that any modules that we need are visible to the engine from the filesystem. This involves using engine.GetSearchPaths() (to see what paths it is looking for) and using engine.SetSearchPaths() to append any additional search paths. But these are from the file system and that is not what I want.

I imagine a nice way might be to receive some kind of an event from the engine like "OnLookingForModuleNamedXYZ(moduleName)". And then I can look up the module in the database or in the DLL's embedded resources and send back the string for the module.

How can I do this?

Answer 1

You'll need to add a custom import hook. Python added a way to register custom import hooks. Prior to this mechanism, you would have to override the built-in __import__ function. But it was particularly error-prone, especially if multiple libraries were both trying to add custom import behavior.

PEP 0302 describes the mechanism in detail.

Essentially, you need to create two objects -- a Finder object and a Loader object.

The Finder object should define a function called find_module

This method will be called with the fully qualified name of the module. If the finder is installed on sys.meta_path , it will receive a second argument, which is None for a top-level module, or package.__path__ for submodules or subpackages [5] . It should return a loader object if the module was found, or None if it wasn't.

If the find_module function finds the module and returns a Loader object, the Loader object should define a load_module function

This method returns the loaded module or raises an exception, preferably ImportError if an existing exception is not being propagated. If load_module() is asked to load a module that it cannot, ImportError is to be raised.

Here is a short example detailing how it would work:

import sys
import imp

class CustomImportFinder(object):

    @staticmethod
    def find_module(fullname, path=None):
        # Do your custom stuff here, look in database, etc.
        code_from_database = "VAR = 1"
        return CustomImportLoader(fullname, path, code_from_database)


class CustomImportLoader(object):

    def __init__(self, fullname, path, code_from_database):
        super(CustomImportLoader, self).__init__()
        self.fullname = fullname
        self.path = path
        self.code_from_database = code_from_database

    def is_package(fullname):
        # is this a package?
        return False

    def load_module(self, fullname):
        code = self.code_from_database
        ispkg = self.is_package(fullname)
        mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
        mod.__file__ = "<%s>" % self.__class__.__name__
        mod.__loader__ = self
        if ispkg:
            mod.__path__ = []
            mod.__package__ = fullname
        else:
            mod.__package__ = fullname.rpartition('.')[0]
        exec(code, mod.__dict__)
        return mod

sys.meta_path.append(CustomImportFinder)

import blah
print(blah)
# <module 'blah' (built-in)>
print(blah.VAR)
# 1