0

I need to make a module that is not in sys.path available to the user. I would like to avoid polluting sys.path with the parent directory of the module to the path, because I don't know what other modules might be there on the user system. Is there a solution that unlike the example below avoids adding the parent directory to the path?

# My library code
import sys
sys.path.insert(0, 'path/to/parent/dir')
import my_module
del sys.path[0]
sys.modules['my_module'] = my_module

# User code
import my_module

Similar questions are available on StackOverflow but don't provide a solution as far as I'm aware:

danijar
  • 32,406
  • 45
  • 166
  • 297
  • Does this answer your question? [What exactly should be set in PYTHONPATH?](https://stackoverflow.com/questions/7850908/what-exactly-should-be-set-in-pythonpath) – Code-Apprentice Nov 29 '21 at 17:35
  • 1
    Who is "the user"? Is this module intended to be used by other programmers in their own programs? If so, then you should package this so they can install it with pip. Then the import will be available for them automagically. – Code-Apprentice Nov 29 '21 at 17:41
  • @Code-Apprentice Thanks, but this is not correct. `sys.path` is initialized from the environment variable `PYTHONPATH` and used for imports and is used for resolving Python imports. The example I gave above works correctly, but will fail if there were unexpected modules inside the directory that I'm adding to `sys.path`. So the link you provided does not answer my question. – danijar Nov 29 '21 at 17:41
  • Thanks for the correction. I did further research to verify my thinking and found out I was incorrect. Comment is already deleted...see my new comment with another question for clarification. – Code-Apprentice Nov 29 '21 at 17:42
  • 2
    I've never had a need to modify `sys.path` directly. What problem are you trying to solve that you feel you need to do this? Is the module you are trying to import not in the project directory? – Code-Apprentice Nov 29 '21 at 17:44
  • @Code-Apprentice Thanks, I'm aware that it could be installed via pip. This is also the most common use case in my library. However, there is another specific use case where the package needs to be imported from the user's filesystem (a parent directory of my library inside the same repository), hence my question how to import the module without polluting `sys.path`. – danijar Nov 29 '21 at 17:44
  • 1
    @Code-Apprentice In summary, I appreciate your help but I'm very clear on the use case and have considered those more common options. I'm an experienced Python programmer and just need the specific question posted above answered :) – danijar Nov 29 '21 at 17:45
  • Sorry if I'm impatient with my comments and repeating myself. I'm thinking through options while waiting for your replies. As far as I know, the options here are creating a separate package, set `PYTHONPATH` or `sys.path` (which now I know are the same thing). I'm not aware of any other way to do what you want. Good luck with finding a solution. – Code-Apprentice Nov 29 '21 at 17:47
  • @Code-Apprentice Thanks! I think it should be possible with `importlib` somehow, perhaps via `importlib.util.find_spec(name, package=None)` or `importlib.machinery.FileFinder` but I haven't been able to get it working yet. – danijar Nov 29 '21 at 17:55
  • To be clear: the intent is that someone else can, with the assistance of your code, specify a path to a module, and have it *dynamically* looked up and imported? Or do you simply want to pre-load modules from *known* locations and give them names that make them available to the client code? – Karl Knechtel Nov 29 '21 at 18:08
  • @KarlKnechtel Is there a big difference between the two? The second option would be sufficient but the first option would be great! In either case, I do not want to add the package parent added to the import path even temporarily to avoid accidentally importing wrong packages. – danijar Nov 29 '21 at 19:02
  • The second option allows you to do the sys.path thing much more straightforwardly and robustly; and also suggests that you ought to be able to just move the files in question somewhere more convenient and avoid the problem entirely. If you really have to avoid polluting the import paths, dynamic import certainly is robust. I wish the standard library were more turn-key about it. – Karl Knechtel Nov 29 '21 at 22:11
  • Yep, the problem is that even if you're extending `sys.path` only for the time for the import, it might break the package that is being imported if there are other modules in its parent directory. Honestly, I think it would be nice to have `import_path()` available as part of the `imp` standard library. – danijar Nov 30 '21 at 16:27

1 Answers1

0

Package

I've created the imptools package to make this solution available more easily:

import imptools  # pip3 install imptools

my_module = imptools.import_path(
    '../path/to/my_module',  # Path to a module directory or single file.
    notfound='error',        # Raise 'error' or 'ignore' if not found.
    reload=False,            # Whether to import if already available.
)

import my_module  # Import statement also works.

Solution

# My library code
import sys
import importlib.util

name = 'my_module'
path = 'path/to/parent/dir'
for finder in sys.meta_path:
  spec = finder.find_spec(name, [path])
  if spec is not None:
    break
else:
  raise ModuleNotFoundError(f'No module named {name!r}', name=name)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)

# User code
import my_module
print(dir(my_module))
danijar
  • 32,406
  • 45
  • 166
  • 297
  • You indicated that `sys.path` pollution is an issue; unless you need the caching provided by `sys.modules` (in which case I'm pretty sure you also need to check explicitly yourself before `importlib.util.module_from_spec`), I would strongly consider returning the module object instead, and letting the user code just call a wrapper from your library and assign the result to `my_module`. Really smells like an `Explicit is better than implicit.` scenario to me. (I happen to have a project of my own that uses a similar "plugin" system.) – Karl Knechtel Nov 29 '21 at 22:15
  • Thanks! Yes, that's what I'm doing in the [imptools](https://github.com/danijar/imptools) package. I'm checking for the module to exist in `sys.modules` and only import it again if `reload=True` is passed. I'm also returning the module object, so it's up to the user whether to use the return value or import the module again. – danijar Nov 30 '21 at 16:25
  • @KarlKnechtel by the way, if you have any other suggestions about handling imports like this, I would be curious to hear about them. Perhaps there are more use cases that would be useful to add to the package. – danijar Nov 30 '21 at 16:31