6

I want to determine the imports of each python module in a library for use in a custom configuration management framework. I've seen tools like snakefood, but I'd prefer to compute a dependency graph during normal program execution, instead of as a ~compilation step.

So far I've tried writing a custom Finder and Loader. Both approaches work as expected the first time a module is imported, but don't trigger on subsequent imports thanks to the sys.modules cache.

I can override __built__.__import__ for notifications every time a module is imported, but it seems like this approach is ill-advised since PEP 302.

Is there an import hook I can place in front of sys.modules cache lookups? Or another way to quickly compute dependencies on the fly?

johnmcs
  • 429
  • 4
  • 10
  • Possible duplicate of [Creating Dependency Graphs in Python](https://stackoverflow.com/questions/4160746/creating-dependency-graphs-in-python) – akshat May 15 '18 at 12:27
  • 2
    This is not a duplicate of the above. That question is looking for a call flow visualisation rather than a dependency graph and its accepted answer reflects that. – Ziyad Edher May 15 '18 at 12:28

1 Answers1

1

It's possible (if hacky) to reassign to sys.modules:

import sys
import inspect

old_sys_modules = sys.modules

class NewSysModules():
  def __getitem__(self, mod_name):
    frame = inspect.currentframe().f_back
    while frame.f_globals["__name__"].startswith("importlib"):
      frame = frame.f_back # go back until we're not in a importlib frame
    importer = frame.f_globals["__name__"]
    print(f"importing {mod_name} from {importer}")

    return old_sys_modules[mod_name]

  def __setitem__(self, mod_name, module):
    old_sys_modules[mod_name] = module

sys.modules = NewSysModules()

However, this might require some maintenance if the import system changes.

internet_user
  • 3,149
  • 1
  • 20
  • 29
  • Thanks! I tweaked this approach slightly (e.g. extended `collections.MutableMapping`) and it seems to work for Python 3. Unfortunately we're on Python 2 until Apache Beam upgrades. If I try to override sys.modules in Python 2, I get notified when `multiprocessing` is imported by `logging`, but I don't get other notifications I would expect. I'll investigate further and report back. – johnmcs May 15 '18 at 16:58