There is a module in my project folder called calendar. Elsewhere in the code, I would like to use the standard library Calendar class. But when I try to import this class, using from calendar import Calendar, this imports from my own module instead, causing errors later.
How can I avoid this? Do I have to rename the module?
It's not necessary to rename the module. Instead, in Python 2.5 and above, use absolute_import to change the importing behavior.
For example, to import the standard library socket module, even if there is a socket.py in the project:
from __future__ import absolute_import
import socket
In Python 3.x, this behaviour is the default. Pylint will complain about the code, but it's perfectly valid.
Actually, solving this is rather easy, but the implementation will always be a bit fragile, because it depends python import mechanism's internals and they are subject to change in future versions.
(the following code shows how to load both local and non-local modules and how they may coexist)
def import_non_local(name, custom_name=None):
import imp, sys
custom_name = custom_name or name
f, pathname, desc = imp.find_module(name, sys.path[1:])
module = imp.load_module(custom_name, f, pathname, desc)
f.close()
return module
# Import non-local module, use a custom name to differentiate it from local
# This name is only used internally for identifying the module. We decide
# the name in the local scope by assigning it to the variable calendar.
calendar = import_non_local('calendar','std_calendar')
# import local module normally, as calendar_local
import calendar as calendar_local
print calendar.Calendar
print calendar_local
The best solution, if possible, is to avoid naming your modules with the same name as standard-library or built-in module names.
The only way to solve this problem is to hijack the internal import machinery yourself. This is not easy, and fraught with peril. You should avoid the grail shaped beacon at all costs because the peril is too perilous.
Rename your module instead.
If you want to learn how to hijack the internal import machinery, here is where you would go about finding out how to do this:
There are sometimes good reasons to get into this peril. The reason you give is not among them. Rename your module.
If you take the perilous path, one problem you will encounter is that when you load a module it ends up with an 'official name' so that Python can avoid ever having to parse the contents of that module ever again. A mapping of the 'official name' of a module to the module object itself can be found in sys.modules.
This means that if you import calendar in one place, whatever module is imported will be thought of as the module with the official name calendar and all other attempts to import calendar anywhere else, including in other code that's part of the main Python library, will get that calendar.
It might be possible to design a customer importer using the imputil module in Python 2.x that caused modules loaded from certain paths to look up the modules they were importing in something other than sys.modules first or something like that. But that's an extremely hairy thing to be doing, and it won't work in Python 3.x anyway.
There is an extremely ugly and horrible thing you can do that does not involve hooking the import mechanism. This is something you should probably not do, but it will likely work. It turns your calendar module into a hybrid of the system calendar module and your calendar module. Thanks to Boaz Yaniv for the skeleton of the function I use. Put this at the beginning of your calendar.py file:
import sys
def copy_in_standard_module_symbols(name, local_module):
import imp
for i in range(0, 100):
random_name = 'random_name_%d' % (i,)
if random_name not in sys.modules:
break
else:
random_name = None
if random_name is None:
raise RuntimeError("Couldn't manufacture an unused module name.")
f, pathname, desc = imp.find_module(name, sys.path[1:])
module = imp.load_module(random_name, f, pathname, desc)
f.close()
del sys.modules[random_name]
for key in module.__dict__:
if not hasattr(local_module, key):
setattr(local_module, key, getattr(module, key))
copy_in_standard_module_symbols('calendar', sys.modules[copy_in_standard_module_symbols.__module__])
In Python 3.5 and up, use the standard library importlib module to import directly from a specified path, bypassing import's lookup mechanism:
import importlib.util
import sys
# For illustrative purposes.
import tokenize
file_path = tokenize.__file__ # returns "/path/to/tokenize.py"
module_name = tokenize.__name__ # returns "tokenize"
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
In actual code, file_path can be set to any path to a .py file to import; module_name should be the name of the module that will be imported (the name that the import system uses to look up the module when further import statements are attempted). Subsequent code will use module as the name of the module; change the variable name module to use a different name.
To load a package instead of a single file, file_path should be the path to the package's root __init__.py.
Absolute import is when Python searches folders in what I will call the system module path (SMP), one at a time, until it finds one that contains the module.
The SMP is created at startup, based on the PYTHONPATH environment variable and some other things. It is represented within Python as a list of strings. After importing the sys standard library module, it is available as sys.path, but it exists whether that module is imported or not. (Usually, Python programmers just call this list "sys.path". However, since we are discussing the import mechanism in deep technical detail, and using modules named sys in examples, it seems appropriate to establish a separate term.)
On the other hand, relative import directly specifies where the module's code should be, relative to the current module, within the context of the current package.
If the standard library module in question is not implemented as a builtin, it may be necessary to set the PYTHONSAFEPATH environment variable before starting Python. This prevents Python from putting the main script's directory (when started like python script.py) or the current working directory (otherwise) at the start of the SMP, as it does by default.
In 3.11, the -P option can be used instead of setting PYTHONSAFEPATH. In 3.4 and up, -I can also be used, but this will also ignore other Python-specific environment variables, and skip adding the per-user site-packages directory to the SMP. (It is also possible, of course, to modify sys.path programmatically to fix the SMP. Unless, of course, the problem is with trying to import sys.)
Once that is taken care of, just use an absolute import.
In 3.x:
import sys # or
from sys import version # or
from sys import * # but beware namespace pollution
In 2.5 through 2.7, a __future__ import is required first:
from __future__ import absolute_import
# proceed as above
In 2.4 and below, a hack is necessary to prevent implicit relative import:
sys = __import__("sys", {})
This should also work in other versions, but is far uglier and more complex than necessary.
Alternately, try hooking into the import machinery to change the lookup behaviour, as described in Omnifarious' answer, Boaz Yaniv's answer or casey's answer. All of these approaches work by emulating (part of) the internal algorithm for searching for modules, but skipping the relative import step and also skipping the first element of sys.path. (The premise is that sys.path is assumed to be "unsafe" - per the above discussion of PYTHONSAFEPATH - and thus using that path would look within the project.)
The standard library generally uses absolute imports. If this causes a problem due to shadowing by the current package, the standard advice is to rename the modules in the current package. Failing that, the same environment variable and command-line flag tricks should work.
In 2.4 and earlier, this will happen by default.
In 2.5 onward, use a relative import, after ensuring that the package is set up correctly.
For the most part, this means making sure that the root of the package is on the SMP. The most robust way to do this is to activate a virtual environment and install the package in that virtual environment. Otherwise, assuming the program starts from some "driver" script, make sure it is in the same folder as the package root. Alternately, run the package (or a subpackage, with the appropriate dotted-path name) as a module, using the -m flag, from the directory that contains the package root.
Assuming the package is set up properly, relative imports look like:
from . import sys # sys.py in the same folder as this source file
from .sys import something # from our source, not the standard library
from .sys import * # again, beware namespace pollution
from .child import sys # child/sys.py
from .. import sys # ../sys.py, IF .. is still within the package root
from ..sibling import sys # ../sibling/sys.py
# Use more .'s to go up more levels first.
# This will not work beyond the package root.
Note that all of these use the from syntax. As explained in the PEP:
Relative imports must always use
from <> import;import <>is always absolute.... because afterimport XXX.YYY.ZZZ...XXX.YYY.ZZZis usable in an expression. But.moduleYis not usable in an expression.
As an absolute last resort, the SMP can be deliberately modified via sys.path, after determining the path to the current file and thus computing the path to the package root, so that absolute imports will work. Please note that this shouldn't ever be necessary in ordinary circumstances. Many popular, heavy-duty Python libraries manage to span hundreds of thousands of lines of code, absolutely none of which mention sys.path in any way.
I'm not aware offhand of any places where the Python standard library uses relative imports internally, and I also can't fathom a good reason to attempt this. That said, it's probably possible by hacking the import machinery. It doesn't sound easy or fun and I'm not about to attempt it here.
See How can I import a module dynamically given the full path?. Brandon Squizzato's answer also summarizes the usual technique.
Originally, there was no special syntax for relative imports. Python would try a relative import first, and then an absolute import if that failed. So for example, code like import sys would import a sys.py from the same folder as the current module - if it existed - rather than the standard library module.
My understanding is that such relative imports would be relative to the package root, not to the current module; but I cannot easily verify this at the moment.
It was apparently possible to work around this, by supplying an empty dict for the globals context to __import__. Quoting:
sys = __import__("sys", {})The
importstatement uses the global namespace to determine which package it is called on; if you pass an empty namespace, it cannot infer package information.
So, by using the __import__ function directly, it was possible to skip the attempted relative import.
PEP 328 introduced a proposal to make Python imports absolute by default, and use relative imports only with explicit syntax. (It also introduced the terms "absolute import" and "relative import" to the Python ecosystem.) Relative imports using this syntax are relative to the module doing the importing; additional leading .s can be used to specify parent packages.
Starting with the very first release of Python 2.5, the new behaviour became available by using a __future__ import:
from __future__ import absolute_import
import math # checks the SMP
On the other hand,
from __future__ import absolute_import
from . import math # imports from the same directory as this source file
or
from __future__ import absolute_import
from .. import math # imports from the immediate parent directory,
# IF it is still within the package root
The behaviour described in PEP 328 became the default. 2.4's implicit relative import is no longer available, and the code explicitly specifies either absolute or relative import - no __future__ import required.
.s used in relative import syntax do not refer to levels in a directory tree, but to levels in a package tree. (After all, there are ways to load modules that don't involve .py files on disk at all!)import this might not find the standard library this even though it's doing an absolute import. (On the other hand, import sys would find the standard library; see the final section.)The last two points can interact in surprising ways:
$ touch token.py
$ python
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help
Type help() for interactive help, or help(object) for help about object.
>>> help()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/_sitebuiltins.py", line 102, in __call__
import pydoc
File "/usr/lib/python3.8/pydoc.py", line 66, in <module>
import inspect
File "/usr/lib/python3.8/inspect.py", line 40, in <module>
import linecache
File "/usr/lib/python3.8/linecache.py", line 11, in <module>
import tokenize
File "/usr/lib/python3.8/tokenize.py", line 35, in <module>
from token import EXACT_TOKEN_TYPES
ImportError: cannot import name 'EXACT_TOKEN_TYPES' from 'token' (/current/working/directory/token.py)
This problem can be avoided by fixing the SMP with any of the techniques from the summary:
$ touch token.py
$ python -I
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help
Type help() for interactive help, or help(object) for help about object.
>>> help()
Welcome to Python 3.8's help utility!
If this is your first time using Python, you should definitely check out
the tutorial on the Internet at https://docs.python.org/3.8/tutorial/.
Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules. To quit this help utility and
return to the interpreter, just type "quit".
To get a list of available modules, keywords, symbols, or topics, type
"modules", "keywords", "symbols", or "topics". Each module also comes
with a one-line summary of what it does; to list the modules whose name
or summary contain a given string such as "spam", type "modules spam".
help>
The module loading system is actually much more complicated than described above. It's deliberately filled with hooks to allow for modifying the behaviour. In particular: aside from the SMP, there is also a "metapath" (made available as sys.meta_path) that contains module-loaders that are actually used to load the modules.
The documentation gives a recipe approximating the process. (Of course, the real thing does not have to solve the bootstrapping problem of importing sys and importlib.util in order to implement the import system!) However, it doesn't really show what each finder in sys.meta_path is doing.
Roughly, by default, for absolute imports:
First, _frozen_importlib.BuiltinImporter checks if the module name matches a module that is built in to Python, so that it can be imported directly. Part of the standard library is implemented this way, and part is not; and that portion has changed over time.
Then, _frozen_importlib.FrozenImporter tries to load a frozen module with the specified name.
Finally, if both of those fail, _frozen_importlib_external.PathFinder searches the SMP.
Because of this, user code might shadow absolute imports of some standard library modules while not shadowing others. Suppose we have this test script import_test.py:
def test_import(name):
module = __import__(name)
return any(attr for attr in dir(module) if not attr.startswith('__'))
if __name__ == '__main__':
print('imported this from standard library?', test_import('this'))
print('imported sys from standard library?', test_import('sys'))
Let's see what happens if empty Python files with those names are added to the CWD first:
$ touch this.py
$ touch sys.py
$ python import_test.py
imported this from standard library? False
imported sys from standard library? True
Since at least 2.0, sys.builtin_module_names lists the names of modules that are built in to Python. Also, since 3.10, sys.stdlib_module_names lists all possible module names in the standard library (even ones that were deliberately excluded when Python was compiled, or which are not available for the current operating system).
I'd like to offer my version, which is a combination of Boaz Yaniv's and Omnifarious's solution. It will import the system version of a module, with two main differences from the previous answers:
Put this somewhere accessible so you can call it (I have mine in my __init__.py file):
class SysModule(object):
pass
def import_non_local(name, local_module=None, path=None, full_name=None, accessor=SysModule()):
import imp, sys, os
path = path or sys.path[1:]
if isinstance(path, basestring):
path = [path]
if '.' in name:
package_name = name.split('.')[0]
f, pathname, desc = imp.find_module(package_name, path)
if pathname not in __path__:
__path__.insert(0, pathname)
imp.load_module(package_name, f, pathname, desc)
v = import_non_local('.'.join(name.split('.')[1:]), None, pathname, name, SysModule())
setattr(accessor, package_name, v)
if local_module:
for key in accessor.__dict__.keys():
setattr(local_module, key, getattr(accessor, key))
return accessor
try:
f, pathname, desc = imp.find_module(name, path)
if pathname not in __path__:
__path__.insert(0, pathname)
module = imp.load_module(name, f, pathname, desc)
setattr(accessor, name, module)
if local_module:
for key in accessor.__dict__.keys():
setattr(local_module, key, getattr(accessor, key))
return module
return accessor
finally:
try:
if f:
f.close()
except:
pass
I wanted to import mysql.connection, but I had a local package already called mysql (the official mysql utilities). So to get the connector from the system mysql package, I replaced this:
import mysql.connector
With this:
import sys
from mysql.utilities import import_non_local # where I put the above function (mysql/utilities/__init__.py)
import_non_local('mysql.connector', sys.modules[__name__])
# This unmodified line further down in the file now works just fine because mysql.connector has actually become part of the namespace
self.db_conn = mysql.connector.connect(**parameters)
