2

I've been caught out by circular imports on a large project. So I'm seeking to find a way to test my code to see which of the modules in the project (and only in the project) are imported when an import statement is run. This is to inform refactoring and make sure there isn't an import somewhere deep within a package that's causing a problem.

Suppose I import project package 'agent', I want to know which project modules also get imported as a result. For instance if 'environment' and 'policy' are imported due to modules deep within the agent package containing those import statements, then I want to see just those listed. So not numpy modules listed for example as they are outside the project and so not relevant for circular dependencies.

So far I have this:

import sys
import agent   # project module

for k, v in sys.modules.items():
    print(f"key: {k}    value: {v}")

example rows:

key: numpy.random   value: <module 'numpy.random' from '/home/robin/Python/anaconda3/envs/rl/lib/python3.9/site-packages/numpy/random/__init__.py'>
key: environment    value: <module 'environment' from '/home/robin/Python/Projects/RL_Sutton/Cliff/environment/__init__.py'>

This does return the modules imported both directly and indirectly but also includes a lot else such as all the components of numpy and builtins etc... If I could filter this dictionary that would solve it.

k is a str, v is <class 'module'>. The module's __str__ method does return the module file path within it so I suppose that could be used but it's not a clean solution. I've tried looking at the documentation for sys.modules and module_type but nothing there gives a way to filter modules to the current project (that I could see).

I tried to modify the solutions for each of these without success:

How to list imported modules?

List imported modules from an imported module in Python 3

ModuleFinder also looked promising but from the limited example I couldn't see how to make path or excludes solve the problem.

Update

I didn't specify this in the original question but I'm importing modules that often look like this:

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    import environment
    import policy

ModuleFinder will find environment and policy even though they won't be imported at runtime and don't matter for cyclic imports. So I adapted the accepted answer below to find only runtime imports.

import agent
import sys

app_dir = '/path/to/projects_folder'

imported_module_names = []

for module_name, mod in sys.modules.items():
    file = getattr(mod, '__file__', '')
    if str(file).startswith(app_dir) and module_name != '__main__':
        imported_module_names.append(module_name)

for module_name in sorted(imported_module_names):
    print(module_name)
Robin Carter
  • 139
  • 9

2 Answers2

0

You can invoke Python with the -v command line option which will print a message each time a module is initialized.

a_guest
  • 34,165
  • 12
  • 64
  • 118
  • Thanks but doesn't solve my problem as the link says this will show "filename or built-in module" and I'm specifically trying to exclude all non-project modules. Also I should perhaps have been clearer that trying to write a test using code. – Robin Carter Feb 08 '21 at 19:24
  • @RobinCarter You can filter the output as necessary, e.g. just `grep`ing for project modules. What kind of test should this be? – a_guest Feb 08 '21 at 19:29
  • I'm trying to write a test that runs in python code within a test package in the project that ensures the project code stays manageable, so with packages not importing things they shouldn't. – Robin Carter Feb 08 '21 at 19:33
  • So to me it sounds like `-v` is the perfect match. Perhaps if you could outline this test (e.g. sketch the implementation in pseudo code, as an edit to your question), then the situation would become clearer? `modulefinder` should work too but without the details it's difficult to say how you want it to be used. – a_guest Feb 08 '21 at 20:07
  • I'm looking for python code not invoking python in a special way. It should be self-contained in a .py file so it can be run like any other python code test. Also filtering the results of -v using grep would be no different to filtering sys.modules.items() as far as I know. Do you see what I'm driving at? If you know how to filter modulefinder for the project modules then that could be a solution but I don't know which is why I'm asking for help. – Robin Carter Feb 08 '21 at 20:29
  • @RobinCarter You can invoke Python from within Python via `subprocess.run([sys.executable, 'path/to/script.py'], capture_output=True, text=True)` and then scan the resulting stdout. Anyway I will add another answer based on modulefinder. – a_guest Feb 08 '21 at 21:04
  • That's interesting. Hope you can see how to solve it with modulefinder. Thanks again. – Robin Carter Feb 08 '21 at 21:14
0

You can use modulefinder to run a script and inspect the imported modules. These can be filtered by using the __file__ attribute (given that you actually import these modules from the file system; don't worry about the dunder attribute, it's for consistency with the builtin module type):

from modulefinder import ModuleFinder

finder = ModuleFinder()
finder.run_script('test.py')

appdir = '/path/to/project'

modules = {name: mod for name, mod in finder.modules.items()
           if mod.__file__ is not None
           and mod.__file__.startswith(appdir)}

for name in modules.keys():
    print(f"{name}")
Robin Carter
  • 139
  • 9
a_guest
  • 34,165
  • 12
  • 64
  • 118
  • I just had to add a minor edit for the None case which needs approval and then I'll mark your answer as accepted. Thanks again. – Robin Carter Feb 08 '21 at 21:40
  • @RobinCarter Correct, this check should be included. – a_guest Feb 08 '21 at 22:00
  • Is there a way to print the modules by the order in which they first started to be imported? The default order seems to be when the first import of a given module was completely. With circular imports, the two are quite different. – ScienceSnake Apr 07 '21 at 23:12
  • @ScienceSnake You can't have top-level cyclic imports in Python, so if you have a cyclic import somewhere else, it's like importing a regular module. Can you give a more detailed example of your problem (perhaps as a new question)? – a_guest Apr 08 '21 at 09:02