0

I have a repo with Python code whose structure could be boiled down to this:

repo_root\
  tool1\
    tool1.py
    tool1_aux_stuff.py
  tool2\
    tool2.py
    tool2_aux_stuff.py
  lib\
    lib1\
      lib1.py
      lib1_aux_stuff.py
    lib2\
      lib2.py
      lib2_aux_stuff.py

The following rules apply to the module usage:

  1. Any tool could use the modules from any library and from its own package, but not from a different tool's one.
  2. Any library could use the modules from any other library, and from its own package. Libraries never access the tool modules.
  3. There must be a way to invoke any tool from any working directory, including those outside repo_root.

The question is: how do I import the lib modules from the tool ones?

I know that if I add __init__.py to each tool and lib directory and to the repo root, then I would be able to use absolute paths from the root, i.e. in tool1.py I could write

import lib.lib1, lib.lib2.lib2_aux_stuff

However, if I execute tool1.py from a random place, e.g.

machine_name: ~/random/place$ python /path/to/repo/tool1/tool1.py

I get the ModuleNotFoundError: No module named 'lib' found error.

I am aware of a workaround which could be implemented using the PYTHONPATH env variable by augmenting it with an absolute path to repo_root and supplying it to the invocation of the tool script, i.e.:

machine_name: ~/random/place$ PYTHONPATH=$PYTHONPATH:/path/to/repo python /path/to/repo/tool1/tool1.py

but I would really prefer something less clunky.

Any ideas how I could do it in a more straightforward way?

Semisonic
  • 1,152
  • 8
  • 21
  • you need to install it, you are asking for stuff that is not in your pythonpath, so you better isntall it in a virtualenv. The problem is that ~/random/place$ python /path/to/repo/tool1/tool1.py has no idea that there is lib.lib1 unless you make it visible to its environment. Just use a virtualenv – E.Serra Mar 28 '19 at 13:43
  • @E.Serra installation was considered, but we ruled it out eventually because the code changes too quickly at this point, and installing the package each time you pull the changes is too cumbersome – Semisonic Mar 28 '19 at 14:14
  • 1
    that does not sound reasonable but as you wish, running python setupy.py install is not cumbersome. – E.Serra Mar 28 '19 at 14:53

1 Answers1

0

Add the path to lib to the scope using sys.path.append('/custom/path/to/modules'). It should then be callable as a module.

You do need to add __init__.py files in any directory that you want to import as if it were a module, otherwise Python doesn't treat them as modules and you'll get another ImportError

Nordle
  • 2,915
  • 3
  • 16
  • 34
  • Ain't explicitly tempering with `sys.path` something that might produce nasty side effects like a module being imported twice? [This answer](https://stackoverflow.com/a/4798648/2520359) suggests it might happen in such case. – Semisonic Mar 28 '19 at 14:16
  • Might do, depends on your use case. The *correct* method to rectify this issue you are having is restructure your project properly to take advantage of how Python imports modules at runtime. Python generally doesn't reload a module when encountering a second import statement for it. Instead, it returns its cached version from `sys.modules` without executing any of its code. However there *are* some pitfalls with this but that's another question in itself! – Nordle Mar 28 '19 at 14:22
  • Could you please elaborate on project restructuring? If that's something that would fix the import issues while not introducing any new ones, I think it would be useful enough for my case and on-topic enough for SO. – Semisonic Mar 28 '19 at 14:44
  • I mean, take advantage of relative imports using the dot notation that python employs, but in order to do this you need to change the structure of your project depending on what you run and from where, and what calls to other scripts they make. There is a plethora of information available, I'd do a quick google and spend some time getting familiar, it'll save you a lot of trouble in the future! – Nordle Mar 28 '19 at 14:51