How can I list all packages/modules available to Python from within a Python script?

Question

I have looked around and do not see a solution for this. What I would like to do is get a list of all packages available in Python at run-time.

I looked at these:

• List all the modules that are part of a python package?
• Python package structure
• How to: get list of modules in a package

But they are not what I am looking for.

I attempted to do this:

import pkgutil
for pkg in pkgutil.walk_packages():
    print(pkg)  # or do something with them...

However, when I do this:

import sys
sys.modules.keys()​​​

It appears that I have loaded all the packages which is not what I want to do, what I want is a list of strings of all packages+modules available to the current Python installation without loading them all when I do it.

Answer 1

Alright, I was curious, and I digged a bit into pkgutil, and I came up with this, which is much simpler than I expected:

list(pkgutil.iter_modules())

It lists all top-level packages/modules available either as regular files or zip packages, without loading them. It will not see other types of packages though, unless they properly register with the pkgutil internals.

Each returned entry is a 3-tuple with the items:

1. module_finder: The file finder instance that found the module
2. name: The name of the module
3. ispkg: A boolean specifying whether it is a regular module or a package.

Example 3-tuple:

(FileFinder('/usr/lib/python3/dist-packages'), 'PIL', True)

And I can confirm that this did not load the PIL package:

In [11]: sys.modules['PIL']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-11-b0fc0af6cc34> in <module>()
----> 1 sys.modules['PIL']

KeyError: 'PIL'

Answer 2

If you need all available modules, not just the ones that are present as files in all the directories in your sys.path, then you can use (the undocumented) pydoc.ModuleScanner (which unfortunately loads the modules to work):

from pydoc import ModuleScanner
import warnings

def scan_modules():
    """Scans for available modules using pydoc.ModuleScanner, taken from help('modules')"""
    modules = {}
    def callback(path, modname, desc, modules=modules):
        if modname and modname[-9:] == ".__init__":
            modname = modname[:-9] + " (package)"
        if modname.find(".") < 0:
            modules[modname] = 1
    def onerror(modname):
        callback(None, modname, None)
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")  # ignore warnings from importing deprecated modules
        ModuleScanner().run(callback, onerror=onerror)
    return modules

modules = list(scan_modules().keys())
print(sorted(modules))

The problem with pkgutil.iter_modules is that it doesn't return all packages, only the ones that are files or directories, but CPython loads a few modules in other special ways that can't be detected by just looking at the files.

The problem with ModuleScanner is that it returns all available modules, including the ones you've pip installed. If you only need the packages that come with Python, then you can start Python with the -I command line option.

Here are the packages that pkgutil doesn't find on Python 3.9:

$ python3.9 -I
Python 3.9.0+ (default, Oct 19 2020, 09:51:18) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> modules = ...  # paste the above code
...
>>> import pkgutil
>>> sorted(set(modules) - {m.name for m in pkgutil.iter_modules()})
['_abc', '_ast', '_bisect', '_blake2', '_codecs', '_collections', '_csv', 
'_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', 
'_locale', '_md5', '_operator', '_peg_parser', '_pickle', 
'_posixsubprocess', '_random', '_sha1', '_sha256', '_sha3', '_sha512', 
'_signal', '_socket', '_sre', '_stat', '_statistics', '_string', 
'_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', 
'_weakref', 'array', 'atexit', 'binascii', 'builtins', 'cmath', 'errno', 
'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 
'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', 'syslog', 'time', 
'unicodedata', 'xxsubtype', 'zlib']

---

This answer is based on @sancho.s's, except instead of parsing the stdout of help('modules'), I have copy/pasted the code that that function runs (you need to copy/paste it because it prints to stdout, there's no helper function):

Answer 3

I put together a very rough way of getting this list (see below), which appears to be more accurate than pkgutil. See details below.

In addition, I found loaded_modules and list-imports, but I tested none of them.

---

I have compared the results of my method with the answer by spectras:

1. All items in the output by spectras (say, modlist2) are in the output here (say, modlist1).
2. There are quite a few items in modlist1 that are not in modlist2. To my surprise, this difference included modules like sys, math, zlib, etc. In my case, the respective lengths were 390 vs. 327, so the method with pkgutil gives quite incomplete results.

---

The method to pull the list of available modules consists of:

1. Capturing output of help into a string
2. Removing spare text from the captured string
3. Splitting multicolumn output

Code is here:

def modules_list() :
    """Return a list of available modules"""
    import sys
    # Capture output of help into a string
    import io
    stdout_sys = sys.stdout
    stdout_capture = io.StringIO()
    sys.stdout = stdout_capture
    help('modules')
    sys.stdout = stdout_sys
    help_out = stdout_capture.getvalue()
    # Remove extra text from string
    help_out = help_out.replace('.', '')
    help_out = help_out.replace('available modules', '%').replace('Enter any module', '%').split('%')[-2]
    # Split multicolumn output
    help_out = help_out.replace('\n', '%').replace(' ', '%').split('%')
    help_out = list(filter(None, help_out))
    help_out.sort()
    return help_out