Handling exceptions raised from imported modules using importlib.import_module

Question

I have a simple plugin system, simplified form below. Idea is that plugins will implement an abstract class and can raise an exception to signal teardown.

# my_plugin.py
import my_app


class MyPlugin(my_app.MyPluginBase):

    def start(self):
        raise my_app.MyLibException()

# my_app.py
import abc
import importlib


class MyLibException(Exception):
    pass


class MyPluginBase(abc.ABC):

    @abc.abstractmethod
    def start(self):
        pass


def main():
    module = importlib.import_module('my_plugin')
    klass = getattr(module, 'MyPlugin')
    try:
        app = klass()
        app.start()
    except MyLibException as e:
        print('ok')
        print(e.__class__)
    except Exception as e:
        print('not ok')
        print(e.__class__)


if __name__ == '__main__':
    main()

Running above results in:

not ok
<class 'my_app.MyLibException'>

What's the correct way of handling exceptions for such a scenario? I would like to catch the raised exception here except MyLibException as e: rather than except Exception as e:.

I'm not sure if it's the correct way, but if you add `from my_app import MyLibException` after defining `MyLibException` in `my_app.py`, the exception is handled and it prints "ok". — Jonathan Feenstra, Oct 12 '19 at 20:11
That looks like some anti-pattern, importing a class within the same file, haven't seen that one before. Is there a similar reference implementation that you may have seen? Thank you — Abhinav Singh, Oct 13 '19 at 01:01
I agree that it looks like an anti-pattern and I haven't found any reference implementations like this. I think the best way to handle it would be to avoid circular dependencies altogether and to define `MyLibException` in a separate file. — Jonathan Feenstra, Oct 13 '19 at 06:53
Agreed! Putting exceptions in a separate module/file will make it work, unfortunately that's not an option I have, as idea here is to have `my_app.py` a single file module extended via `plugins`. However note that, if `my_app.py` starts to directly import `my_plugin.py`, everything work as expected, just that w/ importlib it doesn't. — Abhinav Singh, Oct 14 '19 at 01:41
Are you sure it works when importing `my_plugin` directly? I still have the same problem when I try that. If you really insist on keeping everything in a single file, hopefully my answer provides enough information. However if you are worried about anti-patterns, you should note that circular imports are not a good practice either. — Jonathan Feenstra, Oct 14 '19 at 11:00

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

The root cause of the problem is that MyLibException is recognised as an attribute of __main__ in my_app.py, but when importing it in my_plugin.py it is an attribute of my_app.

The cleanest solution would be to separate MyLibException from the other modules to avoid circular dependencies altogether, but since you mentioned in the comments that this is not an option, the only way I can think of is by importing the exception within the same file. This is obviously not a good practice, but the same can be said about circular imports in general.

Several ways to implement this are:

By adding import my_app (or import __main__ as my_app) to my_app.py and catching the exception using except my_app.MyLibException
By using importlib like you did for importing my_plugin: my_app = importlib.import_module('my_app') and catching the same way as in 1.
from __main__ import MyLibException after defining MyLibException, however this is a violation of PEP8 which states:

Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.

For more suggestions and reasons why this might not be a good idea, see also:

Thanks Jonathan for laying down the options. With option #1 we'll end up with 2 exceptions under different scopes. Also within my_app, I'll have to handle both exceptions, my_lib.MyLibException and MyLibException, as code within my_app too can throw the exception. In long term having separate module sounds like best approach to me, which I am discussing with the community. FWIW, I am looking to solve this for https://github.com/abhinavsingh/proxy.py :) Thank you. — Abhinav Singh, Oct 19 '19 at 16:48

Handling exceptions raised from imported modules using importlib.import_module

1 Answers1