How to trigger a late import of another module only when a certain object is needed from my module?

Question

The Situation

I want to have a module that roughly works like the following:

# my_module.py

my_number = 17

from other_module import foo
my_object = foo(23)

However, there is a problem: Installing other_module causes problems for some users and is only required for those who want to use my_object – which in turn is only a small fraction of users. I want to spare those users who do not need my_object from installing other_module.

I therefore want the import of other_module to happen only if my_object is imported from my_module. With other words, the user should be able to run the following without having installed other_module:

from my_module import my_number

My best solution so far

I could provide my_object via a function that contains the import:

# in my_module.py

def get_my_object():
    from other_module import foo
    my_object = foo(23)
    return my_object

The user would then have to do something like:

from my_module import get_my_object
my_object = get_my_object()

Question

Is there a better way to conditionally trigger the import of other_module? I am mostly interested in keeping things as simple as possible for the users.

Answer 1

I would prefer the get_my_object() approach, but as of Python 3.7, what you ask is possible by defining a module-level __getattr__ function:

# my_module.py

my_number = 17

def __getattr__(name):
    if name != 'my_object':
        raise AttributeError
    global my_object
    from other_module import foo
    my_object = foo(23)
    return my_object

This will attempt to import other_module and call foo only once my_object is accessed. A few caveats:

• It will not trigger for attempts to access my_object by global variable lookup within my_module. It will only trigger on my_module.my_object attribute access, or a from my_module import my_object import (which performs attribute access under the hood).
• If you forget to assign to the global my_object name in __getattr__, my_object will be recomputed on every access.

• Module-level __getattr__ does nothing before Python 3.7, so you may want to perform a version check and do something else for Python 3.6 and below:

  import sys
  if sys.version_info >= (3, 7):
      def __getattr__(name):
          ...
  else:
      # Do something appropriate. Maybe raise an error. Maybe unconditionally
      # import other_module and compute my_object up front.
  

Answer 2

Approach A – clean solution

Create a new separate module and have the user import the object from the other module. For example

from my_module import my_number # regular use
from my_module.extras import my_object # for a small part of the user base

This means in your code you create a module folder my_module with an __init__.py where you import the usual stuff and don't import the extras submodule.

If you don't want to put extras in my_module (simpler), just create my_object in an individual extras.py module.

Approach B – signals bad architecture 99% of times

You can use importlib.import_module to dynamically import a module inside get_my_object without polluting the global space and that is cleaner than an import inside a function which creates side effects such as overriding your global variable with that import name (see this question and answers), however this is usually a sign of bad coding patterns on other part of the code.

Approach C – simple and effective

I usually tend to favour this simple pattern when there are users that might not have a library, as Python 3 discourages imports that are not at top level:

try:
    import other_module
    _HAS_OTHER_MODULE_ = True
except:
    _HAS_OTHER_MODULE_ = False


def get_my_object():
    assert _HAS_OTHER_MODULE_, "Please install other module"
    return other_module.foo(23)

Answer 3

This is a common hack to be done:

import sys

class MockModule:
    def __init__(self, module):
        self.module = module

    def __getattr__(self, attr):
        if attr == 'dependency_required_var':
            try:
                import foo
                return self.module.dependency_required_var
            except ImportError:
                raise Exception('foo library is required to use dependency_required_var')
        else:
            return getattr(self.module, attr)


MockModule.__name__ = __name__
sys.modules[__name__] = MockModule(sys.modules[__name__])

dependency_required_var = 0

With this PEP, we can simply do (we should be able to but I couldn't get it to work) the following in Python 3.7 and higher:

def __getattr__(attr):
    if attr == 'dependency_required_var':
        try:
            import foo
            return dependency_required_var
        except ImportError:
            raise Exception('foo library is required to use dependency_required_var')
    else:
        return globals()[attr]

The PEP seems to be accepted, but the relevant pull request from the PEP seems to be closed, I'm actually not sure if it has been implemented or not.