How to determine if a module name is part of python standard library

Question

I have a module name as a string (e.g. 'logging') that was given by querying the module attribute of an object.

How can I differentiate between modules that are part of my project and modules that are part of python standard library?

I know that I can check if this module was installed by pip using pip.get_installed_distributions(), but these are not related to the standard library

Note: I'm working on python 2.7 so solutions that are valid only in python 3.x are less relevant.

Unlike the answer here, I was looking for a solution that can be run in O(1) and will not require holding an array of results nor having to scan the directory for every query.

Thanks.

Possible duplicate of [How can I get a list of all the Python standard library modules](https://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules) — Alperen, Sep 27 '17 at 07:25
I'm curious to know why you want to do this in case there's some other way to solve what you're trying to do. — Stephen Paulger, Sep 27 '17 at 08:01
As a rule, you don't want to do that. E.g. when the codebase will be run by a different Python version, some modules may move between being stock and 3rd-party/backports. If your problem is that your import syntax is ambiguous, then make it unambiguous instead e.g. by fully qualifying the modules or with relative imports. — ivan_pozdeev, Sep 27 '17 at 09:09
Also looking at the duplicate, this answer with isort might be better, at least for me I already had it installed instead of stdlib_list. https://stackoverflow.com/a/21659703/3387223 — CodeMonkey, Mar 08 '23 at 09:48

score 4 · Accepted Answer · edited Jun 17 '19 at 06:21

4

Quick 'n dirty solution, using the standard module imp:

import imp
import os.path
import sys

python_path = os.path.dirname(sys.executable)

my_mod_name = 'logging'

module_path = imp.find_module(my_mod_name)[1]
if 'site-packages' in module_path or python_path in module_path or not imp.is_builtin(my_mod_name):
    print('module', my_mod_name, 'is not included in standard python library')

edited Jun 17 '19 at 06:21

laike9m

18,344
20
107
140

answered Sep 27 '17 at 07:15

Guillaume

5,497
3
24
42

1

As an example of ways in which this fails, it won't detect modules imported from the script's directory. "Quick and dirty" is an apt description. – user2357112 Sep 27 '17 at 07:20
1

I know. "Quick 'n dirty" is there for a reason ;) I added a refinement to include modules imported outside python install folder. – Guillaume Sep 27 '17 at 07:23
@user2357112, do you have other examples of how this may fail? Guillaume's update seems to fix this issue. – Yohai Devir Sep 27 '17 at 07:26
@YohaiDe: The update fixes that problem, but breaks for C modules like `sys` or `itertools`. – user2357112 Sep 27 '17 at 07:32
1

fixed for C Builtin modules – Guillaume Sep 27 '17 at 07:44

Alperen · Answer 2 · 2023-03-23T10:46:07.557

4

EDIT:

I used the solution (source).

import distutils.sysconfig as sysconfig
import os

def std_modules():
    ret_list = []
    std_lib = sysconfig.get_python_lib(standard_lib=True)
    for top, dirs, files in os.walk(std_lib):
        for nm in files:
            if nm != '__init__.py' and nm[-3:] == '.py':
                ret_list.append(os.path.join(top, nm)[len(std_lib)+1:-3].replace('\\','.'))
    return ret_list
            
l = std_modules()
print("logging" in l)
print("os" in l)

Output:

False
True

This works in both Python 2 and Python 3.

BEFORE EDIT:

I guess, you can use Python Docs. Here are standard library parts of Python 2 Docs and Python 3 Docs. Also, you can select the exact version of Python.

edited Mar 23 '23 at 10:46

answered Sep 27 '17 at 07:16

Alperen

3,772
3
27
49

I need an method that can work on the fly. I can save an offline list based on this list, but it'll need non trivial updates. – Yohai Devir Sep 27 '17 at 07:18
@YohaiDe OK, try this one. – Alperen Sep 27 '17 at 07:32

smarie · Answer 3 · 2020-09-01T10:55:56.773

2

There is also a good answer in an old duplicate of this post: https://stackoverflow.com/a/28873415/7262247

This is how you can do it (you need to pip install stdlib_list first):

from stdlib_list import stdlib_list
import sys

all_stdlib_symbols = stdlib_list('.'.join([str(v) for v in sys.version_info[0:2]]))

module_name = 'collections'

if module_name in all_stdlib_symbols:
    print("%s is in stdlib" % module_name)

edited Sep 01 '20 at 10:55

answered Jul 03 '19 at 16:30

smarie

4,568
24
39

You need `pip install stdlib_list` first. – shellcat_zero Aug 31 '20 at 20:51
indeed thanks @shellcat_zero, I updated the post accordingly. – smarie Sep 01 '20 at 10:56

score 2 · Answer 4 · answered May 13 '20 at 17:14

None of the solutions above were what I wanted, so I did it yet another way. Posting here in case it's useful to anyone.

import os

def standard_lib_names_gen(include_underscored=False):
    standard_lib_dir = os.path.dirname(os.__file__)
    for filename in os.listdir(standard_lib_dir):
        if not include_underscored and filename.startswith('_'):
            continue
        filepath = os.path.join(standard_lib_dir, filename)
        name, ext = os.path.splitext(filename)
        if filename.endswith('.py') and os.path.isfile(filepath):
            if str.isidentifier(name):
                yield name
        elif os.path.isdir(filepath) and '__init__.py' in os.listdir(filepath):
            yield name

>>> standard_lib_names = set(standard_lib_names_gen(include_underscored=True))
>>> # verify that a few known libs are there (including three folders and three py files)
>>> assert {'collections', 'asyncio', 'os', 'dis', '__future__'}.issubset(standard_lib_names)
>>> # verify that other decoys are not in there
>>> assert {'__pycache__', 'LICENSE.txt', 'config-3.8-darwin', '.DS_Store'}.isdisjoint(standard_lib_names)
>>>
>>> len(standard_lib_names)
200
>>>
>>> # not including underscored
>>> standard_lib_names = set(standard_lib_names_gen(include_underscored=False))
>>> len(standard_lib_names)
184

How to determine if a module name is part of python standard library

4 Answers4

Linked