2

Background

I have a package i'm working on which i have also installed into site-packages. (I've installed my own package using pip)

Question

How does Python know to import from the "local" code (the code i'm working on), rather than importing from site-packages? I'm asking because i want to better understand and i'm worried that somehow i might import code from site-packages when i really want to use my own (latest) code

It’s confusing to me that this is not more clear in how we write imports. It seems to me that the programmer doesn’t know if she is importing her own code or importing from site-packages. Am i missing something here?

One thought i had was to use relative imports but if i understand they are not recommended in general

I'm using Python 3.8.5

sunyata
  • 1,843
  • 5
  • 27
  • 41
  • 1
    Did you install your own package that you are currently working on using `pip -e .` in the root of your own project (with your `setup.py`)? If so it should be the only one being used. Naturally do not install the published package that you may have put on PyPI. – metatoaster Feb 20 '21 at 10:19
  • @metatoaster Thank you for your comment, i didn't know about `pip -e`! Okay i didn't know it was bad practice to install my own package like i've done, live and learn :) I'm guessing the reason this is a bad idea is because of the issue i brought up in the question? – sunyata Feb 21 '21 at 13:22
  • 1
    It depends on what you specifically want your package for the particular environment. If you want a use case where any changes to your code is reflected immediately in the environment, the editable install (i.e. `pip -e`) is the way to go (which seems to be what you are after here). Otherwise the standard installation method is to build a package with a frozen copy, which `pip` will then install into `site-packages`; useful for seeing what the final install may look like, but not very easily editable (basically involves a reinstallation of the package), which was what you may have done. – metatoaster Feb 21 '21 at 13:38

3 Answers3

3

Python's import mechanism (commonly referred as the import machinery) is based on Finders and Loaders. When you import a module using the import statement, a series are finders try to find the module that you are importing. You can see the list of all finders that are triggered in order by:

>>> import sys
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>,
 <class '_frozen_importlib.FrozenImporter'>,
 <class '_frozen_importlib_external.PathFinder'>]

If none of the finders are able to resolve the module, then ModuleNotFoundError is raised.

BuiltinImporter resolves modules like sys, time, gc etc which are preloaded into the interpreter. FrozenImporter resolves frozen modules. PathFinder resolves modules by going through the paths in sys.path.

Printing sys.path shows that your current directory is the first entry and hence is searched for a match first.

Here's a simple sequence you can use to reason about where the import is going to happen from:

  1. Is the module part of built-in modules. Example: import sys. Even if you have a file named sys.py in your current directory, since BuiltinImporter is called first, it will resolve the built-in sys module.
  2. Is the module present in your current directory, since that is the first entry in sys.path.
  3. Is the module available as part of core python or site-packages.

Modules that are imported by PathFinder (ones present in sys.path) will have __path__ as an attribute that you can check to see where it is located.

It is also possible to reorder the finders in sys.meta_path or even add your own custom finders and loaders giving you full control and creativity to modify the default machinery. Custom finders and loaders can be registered to look for modules elsewhere in the system or even dynamically from external systems.

sanygeek
  • 316
  • 2
  • 5
  • Thank you very much for your answer, this really helps clarify how things work under the hood, now i can rest easy, having a better understanding of how the import order works! --- PS: At first i was confused because i could not see the present directory when i was looking at sys.path, but this seems to be because i was using the "python console" inside pycharm (my IDE). When i use a terminal (separate from pycharm) instead and go into the virtual environment and look at sys.path i can see that the first entry is the empty string (so the current directory) – sunyata Feb 21 '21 at 13:33
0

If you want to symlink your package as development install, use:

pip install --no-dependencies --editable .

or:

python setup.py develop

No need to modify sys.path

Your current folder will be symlinked to site-packages. Therefore, if your package name is foo, you can import modules with

from foo import bar
# or
import foo
foo.bar.do_something()
Alex
  • 2,784
  • 2
  • 32
  • 46
0

I would consider creating a virtual enviroment which doesnt have the pip package installed. This was you can be sure you are using your local version!

If you are confused about where a package is imported from you can use .__path__ to find out.

import my_package
print(my_pacakge.__path__)
Jack McKay Fletcher
  • 129
  • 8
  • According to the orinial post, he is already using `venv`. I am not using venv, but I believe you still need to install the package symlinked (`--editable`) in the venv to use/test it. – Alex Feb 20 '21 at 11:29