59

I noticed Flask was using Werkzeug to __import__ a module, and I was a little confused. I went and checked out the docs on it and saw that it seems to give you more control somehow in terms of where it looks for the module, but I'm not sure exactly how and I have zero idea how it's different from importlib.import_module.

The odd thing in the Werkzeug example is that it just says __import__(import_name), so I don't see how that's any different from just using the import statement, since it's ignoring the optional extra parameters.

Can anyone explain? I looked at other people having asked similar questions on SO previously but they weren't very clearly phrased questions and the answers didn't address this at all.

temporary_user_name
  • 35,956
  • 47
  • 141
  • 220
  • 2
    Use `__import__` where `you want to import a module whose name is only known at runtime`. Other than this... as you said this gives your more power compared to import – sarveshseri Jan 30 '15 at 08:18
  • If memory serves correctly there is a David Beasely video on import, within the video he mentions that WerkZeug overloaded import for some reason. – Carel Dec 28 '16 at 00:24

2 Answers2

74

__import__ is a low-level hook function that's used to import modules; it can be used to import a module dynamically by giving the module name to import as a variable, something the import statement won't let you do.

importlib.import_module() is a wrapper around that hook* to produce a nice API for the functionality; it is a very recent addition to Python 2, and has been more fleshed out in Python 3. Codebases that use __import__ generally do so because they want to remain compatible with older Python 2 releases, e.g. anything before Python 2.7.

One side-effect of using __import__ can be that it returns the imported module and doesn't add anything to the namespace; you can import with it without having then to delete the new name if you didn't want that new name; using import somename will add somename to your namespace, but __import__('somename') instead returns the imported module, which you can then ignore. Werkzeug uses the hook for that reason in one location.

All other uses are to do with dynamic imports. Werkzeug supports Python 2.6 still so cannot use importlib.

* importlib is a Pure-Python implementation, and import_module() will use that implementation, whist __import__ will use a C-optimised version. Both versions call back to importlib._bootstrap._find_and_load() so the difference is mostly academic.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • 1
    I don't quite understand what you mean by "you can import with it without having then to delete the new name if you didn't want that new name." If you have a minute could you maybe flesh that out as if you were talking to a toddler? – temporary_user_name Jan 30 '15 at 09:09
  • 3
    @Aerovistae: updated that a little; `import sys` adds the name `sys` to your namespace (bound to the module), `__import__('sys')` returns the `sys` module object to the caller, which is then free to ignore that object. – Martijn Pieters Jan 30 '15 at 09:33
  • 1
    @MartijnPieters By namespace do you mean `sys.modules`? – NeoWang Aug 06 '15 at 01:43
  • 1
    @NeoWang: by namespace I mean the globals of a module. These are stored as a dictionary on the `module` instance, which in turn is stored in `sys.modules`. – Martijn Pieters Aug 06 '15 at 08:33
  • (Modified link in this comment) Does this answer requires modification? According to [this](https://stackoverflow.com/a/27307159/1317018) (specifically [this](https://docs.python.org/3/whatsnew/3.3.html#using-importlib-as-the-implementation-of-import)), `__import__()` is not `importlib.__import__` and `import` statement is not implemented by `importlib`, that is, I guess `import` statement is now `importlib.import_module()`, right? – Mahesha999 Feb 21 '18 at 10:21
  • @Mahesha999: `import` is still using `__import__`, see [`import_name()` in the Python evaluation loop](https://github.com/python/cpython/blob/v3.6.4/Python/ceval.c#L5199-L5237) (which the `IMPORT_NAME` bytecode calls). The [`__import__()` built-in](https://github.com/python/cpython/blob/v3.6.4/Python/bltinmodule.c#L227-L257) calls the [`PyImport_ImportModuleLevelObject()` function](https://github.com/python/cpython/blob/v3.6.4/Python/import.c#L1480-L1645), which is a C port of the `importlib._bootstrap.__import__()` and its dependencies. – Martijn Pieters Feb 21 '18 at 12:21
  • @Mahesha999: technically speaking, `importlib.import_module()` goes straight to `importlib._bootstrap._gcd_import()`, which is one of the helper functions that `importlib.__import__` uses, and which the C implementation has optimised; both codepaths end up calling `importlib._bootstrap._find_and_load()` though. *At no point does `import` use `importlib.import_module()`*. – Martijn Pieters Feb 21 '18 at 12:22
  • Thats some more implementational facts and makes things more confusing. However, now I am trying to guess what all things I should keep in mind while deciding what I should be using: `import`, `__import__` and `importlib.__import__`. For example, now I am trying to write a code which will take a string of multiple py files (stored in database, instead of file system, for security reasons, uploaded by user from browser), create module from them and properly handle import in them. Or in any other scenario, what to prefer when: `import` statement, `__import__` or `importlib.__import__`. – Mahesha999 Feb 22 '18 at 05:58
  • 2
    @Mahesha999: To dynamically produce a module from source code stored in a string, you don't need to look at `__import__` at all. Create an empty module instance with `module = imporlib.util.module_from_spec(importlib.machinery.ModuleSpec(module_name, None, origin=''))`, store the module with `sys.modules[module_name] = module`, then use `exec(sourcecode, module.__dict__)` to execute your source code. And don't ever execute user-supplied code, that is extremely insecure. – Martijn Pieters Feb 22 '18 at 08:24
  • @MartijnPieters the issue was that what if that module itself have `import module1`, where `module1` is also a user supplied module (we have already considered security stuff)? So, I guess I had to override `__import__`, right? I have done `module=types.ModuleType(__name__)` instead of `module = imporlib.util.module_from_spec(...)` inside overrided `__import__`, is that ok? – Mahesha999 Feb 22 '18 at 09:10
  • @Mahesha999: No, then you are going too low level. Add a custom [`MetaPathFinder`](https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder) to [`sys.meta_path`](https://docs.python.org/3/library/sys.html#sys.meta_path), and Python will ask it how to load your modules (Python asks each object in `sys.meta_path` in order for a `find_spec()` result until a result is found). This lets you specify a custom loader, so you can load your module from a database. – Martijn Pieters Feb 22 '18 at 09:27
  • [Here](https://docs.python.org/3.8/reference/import.html#replacing-the-standard-import-system), found these statements: *If it is acceptable to only alter the behaviour of import statements without affecting other APIs that access the import system, then replacing the builtin `__import__()` function may be sufficient. This technique may also be employed at the module level to only alter the behaviour of import statements within that module.* Does that suggest overriding `__import__` is suited for my scenario of dynamically creating and importing modules from source string? – Mahesha999 Feb 22 '18 at 12:27
  • @Mahesha999: this is really all getting way too long. If you feel your use case is better served by setting `builtins.__import__` to your own hook (or by setting it locally for a module executed with `exec`, like `namespace = {'__import__': your_hook}` then exec(sourcecode, namespace)`) then use that. I don't know your exact use case (nor are the comments a place for such a discussion). – Martijn Pieters Feb 22 '18 at 18:32
  • yeah I know this is getting too long...very sorry and super thanks for pointing me to meta path stuff...will explore more myself and prepare question if I get stuck anywhere...thanks again!!! – Mahesha999 Feb 22 '18 at 19:20
  • is there a way to use __import__(modulename) to not only return the module but also import it? I want a dual-solution, instead of having a place where I can check the module exists in the system and then load it into the namespace. – Vaidøtas I. Mar 24 '20 at 11:04
  • @VaidøtasIvøška: `__import__(modulename)` *does* import it, if not already imported. Not sure what you are asking here. – Martijn Pieters Mar 24 '20 at 11:06
  • @MartijnPieters I am confused - I am running a loop on a list of module names and using __import__(next(module_str_names)) inside the loop, but if I do not declare the module name in an e.x. 'import csv' after, outside of the loop - I get a NameError, which tells me no module was imported, they were only checked. :/ – Vaidøtas I. Mar 24 '20 at 11:10
  • @VaidøtasIvøška: the issue is that you never assign the object returned by `__import__` to a name. If your code is relying on the the `csv` module being imported under the name `csv`, then **just use `import csv`**. Why use dynamic imports when the *rest* of your code is not dynamic? If you do need to import modules dynamically, use a dictionary or list to store the imported modules. Also see [How do I create a variable number of variables?](https://stackoverflow.com/q/1373164) – Martijn Pieters Mar 24 '20 at 11:12
  • @VaidøtasIvøška: The `import` statement is a binding action, just like name assignment is. So `import csv` is equivalent to `csv = __import__('csv')`, while `import csv as foo` is equivalent to `foo = __import__('csv')`, `from csv import reader` is equivalent to `reader = __import__('csv').reader`, etc. You are not assigning to names in your loop. – Martijn Pieters Mar 24 '20 at 11:14
  • @MartijnPieters my point is to check for their existence in the system as dependencies. If I assign the __import__(modulename) to like 'module = __import__(modulename)' and in the same loop do: 'import module' then I get a different problem of an infinite loop: 'csv is not present in the system. Installing with pip... ERROR: Could not find a version that satisfies the requirement csv (from versions: none) ERROR: No matching distribution found for csv – Vaidøtas I. Mar 24 '20 at 11:16
  • @VaidøtasIvøška: `csv` is a [**standard library module**](https://docs.python.org/3/library/csv.html) and doesn't need installing. Or testing for. It is always there if Python is installed. Your code should not be responsible for testing dependencies, your installer should be. If you have optional dependencies, just use `try: import modulename`, `except ImportError: `. – Martijn Pieters Mar 24 '20 at 11:17
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/210222/discussion-between-vaidotas-ivoska-and-martijn-pieters). – Vaidøtas I. Mar 24 '20 at 11:18
9

__import__(import_name), so I don't see how that's any different from just using the import statement

Both __import__() and importlib.import_module() allow you to import a module when you have the module name as a string. You cannot write:

x = 're'
import x

or you'll get:

 File "1.py", line 3, in <module>
 import x ImportError: No module named x
7stud
  • 46,922
  • 14
  • 101
  • 127