Importing module with "current-directory" imports

Question

I have the following code to load a module dynamically:

def load_module(absolute_path):
    import importlib.util
    module_name, _ = os.path.splitext(os.path.split(absolute_path)[-1])
    try:
        py_mod = imp.load_source(module_name, absolute_path)
    except ImportError:
        module_root = os.path.dirname(absolute_path)
        print("Could not directly load module, including dir: {}".format(module_root))
        spec = importlib.util.spec_from_file_location(
            module_name, absolute_path, submodule_search_locations=[module_root, "."])
        py_mod = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(py_mod)
    return py_mod

It works really well, except for the case when it is trying to import a script in the same folder (and not part of a package with the same name). For example, script a.py is doing import b. It results in the error ImportError: No module named 'b' (which is common for Python 3).

But I would really like to find a way to solve this? It would be solved by prepending:

import sys
sys.path.append(".")

to script "a".

Though I hoped it could have been solved by:

submodule_search_locations=[module_root, "."]

Oh yea, the rationale is that I also want to support importing modules that are not proper packages/modules, but just some scripts that would work in an interpreter.

Reproducible code:

~/example/a.py

import b

~/example/b.py

print("hi")

~/somewhere_else/main.py (located different from a/b)

import sys, os, imp, importlib.util

def load_module(absolute_path) ...

load_module(sys.argv[1])

Then run on command line:

cd ~/somewhere_else
python3.5 main.py /home/me/example/a.py

which results in ImportError: No module named 'b'

The following code solves it, but of course we cannot put sys.path stuff manually in all scripts.

~/example/a.py (2)

import sys
sys.path.append(".")
import b

I really hope others might have a solution that I didn't think of yet.

Appendix

def load_module(absolute_path):
    import importlib.util
    module_name, _ = os.path.splitext(os.path.split(absolute_path)[-1])
    try:
        py_mod = imp.load_source(module_name, absolute_path)
    except ImportError as e:
        if "No module named" not in e.msg:
            raise e

        missing_module = e.name
        module_root = os.path.dirname(absolute_path)

        if missing_module + ".py" not in os.listdir(module_root):
            msg = "Could not find '{}' in '{}'"
            raise ImportError(msg.format(missing_module, module_root))

        print("Could not directly load module, including dir: {}".format(module_root))
        sys.path.append(module_root)
        spec = importlib.util.spec_from_file_location(module_name, absolute_path)
        py_mod = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(py_mod)
    return py_mod

Please don't cut-and-run; deleting then reposting to evade votes your received. Yes, you need to manipulate `sys.path` if you want to import from the same directory but not have a package. — Martijn Pieters, Aug 06 '16 at 12:17
@MartijnPieters Having to wait to get it off-hold was a bit silly given that I gave it a huge revamp. — PascalVKooten, Aug 06 '16 at 12:35
That's what the review queue is for. Deleting and reposting like that can and will land you in an (automated) question ban. — Martijn Pieters, Aug 06 '16 at 12:36

score 5 · Accepted Answer · answered Aug 06 '16 at 12:36

5

It doesn't really matter that this you are using dynamic importing here; the same problem applies to any code making assumptions about the current directory being on the path. It is the responsibility of those scripts to ensure that the current directory is on the path themselves.

Rather than use '.' (current working directory), use the __file__ global to add the directory to the path:

import os.path
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)

You could retry your dynamic import by adding os.path.dirname(absolute_path) to sys.path when you have an ImportError (perhaps detecting it was a transient import that failed), but that's a huge leap to make as you can't distinguish between a missing dependency and a module making an assumption about sys.path.

answered Aug 06 '16 at 12:36

Martijn Pieters

1,048,767
296
4,058
3,343

Perhaps though as a safe-guard it is possible to scan whether that folder (in my script called `module_root`) contains a file with the name which does not exist (as pointed out by your link). That should be really safe; otherwise the script would not run anyway. I had not realised I should add just add `sys.path.append(module_root)` before doing that command (but only if file of that name exists). So thanks a lot, it really did clarify. It also answered the question why I'm able to do such imports in an interpreter: Emacs adds the current directory to path. – PascalVKooten Aug 06 '16 at 12:42
The directory a script is defined in is also auto-added to `sys.path` when running scripts, but it's not something I'd rely on. – Martijn Pieters Aug 06 '16 at 12:44
Yea, but I expect this to be ran from command-line, thus from a different directory. I patched it carefully, see appendix. – PascalVKooten Aug 06 '16 at 12:52

score 0 · Answer 2 · answered Feb 17 '21 at 23:16

In my case, what worked was importlib.import_module('file_basename') — without any package specified. See below trial & error & success:

>>> importlib.import_module('.trainer', '.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named '.'

>>> importlib.import_module('.trainer', None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/importlib/__init__.py", line 122, in import_module
    raise TypeError(msg.format(name))
TypeError: the 'package' argument is required to perform a relative import for '.trainer'

>>> importlib.import_module('.trainer', '')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/importlib/__init__.py", line 122, in import_module
    raise TypeError(msg.format(name))
TypeError: the 'package' argument is required to perform a relative import for '.trainer'

>>> importlib.import_module('trainer')
<module 'trainer' from '/home/ubuntu/project1/trainer.py'>