ModuleNotFoundError: No module named 'ansible_collections.ansible.builtin'

Question

I'm trying to write a python script that executes a local ansible playbook using the ansible.executor.playbook_executor.PlaybookExecutor class like the second answer of this question. I'm using python 3.11.4

The script looks like this:

from ansible import context
from ansible.module_utils.common.collections import ImmutableDict
from ansible.executor.playbook_executor import PlaybookExecutor
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager
from ansible.vars.manager import VariableManager


loader = DataLoader()

context.CLIARGS = ImmutableDict(tags={}, listtags=False, listtasks=False, listhosts=False, syntax=False, connection='ssh',
                                module_path=None, forks=100, remote_user='bitv', private_key_file="/Users/an/.ssh/ssh_rsa_keys",
                                ssh_common_args=None, ssh_extra_args=None, sftp_extra_args=None, scp_extra_args=None, become=True,
                                become_method='sudo', become_user='root', verbosity=True, check=False, start_at_task=None)

inventory = InventoryManager(loader=loader, sources='/Users/an/Documents/Repositories/dlamp-stack/inventories/simple_lamp_stack/ansible01')

variable_manager = VariableManager(loader=loader, inventory=inventory)

pbex = PlaybookExecutor(playbooks=['/Users/an/Documents/Repositories/dlamp-stack/test.yml'], inventory=inventory, variable_manager=variable_manager, loader=loader, passwords={})

results = pbex.run()

and I get the following error when executing:

[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details
Traceback (most recent call last):
  File "/Users/an/Documents/Projekte/ansible-java-client/src/main/playbook-exec.py", line 18, in <module>
    results = pbex.run()
              ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/executor/playbook_executor.py", line 111, in run
    pb = Playbook.load(playbook_path, variable_manager=self._variable_manager, loader=self._loader)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/__init__.py", line 52, in load
    pb._load_playbook_data(file_name=file_name, variable_manager=variable_manager)
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/__init__.py", line 107, in _load_playbook_data
    entry_obj = Play.load(entry, variable_manager=variable_manager, loader=self._loader, vars=vars)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/play.py", line 157, in load
    return p.load_data(data, variable_manager=variable_manager, loader=loader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/base.py", line 171, in load_data
    setattr(self, name, method(name, ds[name]))
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/play.py", line 198, in _load_pre_tasks
    return load_list_of_blocks(ds=ds, play=self, variable_manager=self._variable_manager, loader=self._loader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/helpers.py", line 68, in load_list_of_blocks
    Block.load(
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/block.py", line 92, in load
    return b.load_data(data, variable_manager=variable_manager, loader=loader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/base.py", line 171, in load_data
    setattr(self, name, method(name, ds[name]))
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/block.py", line 120, in _load_block
    return load_list_of_tasks(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/helpers.py", line 323, in load_list_of_tasks
    t = Task.load(task_ds, block=block, role=role, task_include=task_include, variable_manager=variable_manager, loader=loader)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/task.py", line 136, in load
    return t.load_data(data, variable_manager=variable_manager, loader=loader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/base.py", line 161, in load_data
    ds = self.preprocess_data(ds)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/playbook/task.py", line 204, in preprocess_data
    (action, args, delegate_to) = args_parser.parse()
                                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/parsing/mod_args.py", line 307, in parse
    context = action_loader.find_plugin_with_context(item, collection_list=self._collection_list)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/plugins/loader.py", line 586, in find_plugin_with_context
    result = self._resolve_plugin_step(name, mod_type, ignore_deprecated, check_aliases, collection_list, plugin_load_context=plugin_load_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/plugins/loader.py", line 682, in _resolve_plugin_step
    return self._find_plugin_legacy(name, plugin_load_context, ignore_deprecated, check_aliases, suffix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/plugins/loader.py", line 787, in _find_plugin_legacy
    return self._find_fq_plugin(fq_name=candidate_fqcr, extension=suffix, plugin_load_context=plugin_load_context, ignore_deprecated=ignore_deprecated)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/plugins/loader.py", line 476, in _find_fq_plugin
    routing_metadata = self._query_collection_routing_meta(acr, plugin_type, extension=extension)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansible/plugins/loader.py", line 434, in _query_collection_routing_meta
    collection_pkg = import_module(acr.n_python_collection_package_name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ansible_collections.ansible.builtin'

And these python packages are installed:

an@lpt02 ansible % pip3 list
Package        Version
-------------- -------
ansible        8.2.0
ansible-core   2.15.2
ansible-runner 2.3.3
cffi           1.15.1
cryptography   41.0.2
docutils       0.20.1
Jinja2         3.1.2
lockfile       0.12.2
MarkupSafe     2.1.3
packaging      23.1
pexpect        4.8.0
pip            23.2.1
ptyprocess     0.7.0
pycparser      2.21
python-daemon  3.0.1
PyYAML         6.0
resolvelib     1.0.1
setuptools     67.6.1
six            1.16.0
wheel          0.40.0

There is no ansible_collections package listed so I went to /usr/local/lib/python3.11/site-packages and found it there: drwxr-xr-x 52 an admin 1664 28 Jul 16:02 ansible_collections.

Then in /usr/local/lib/python3.11/site-packages/ansible_collections/ansible (following ansible_collections.ansible.builtin) there is no builtin, probably resulting in the ModuleNotFoundError:

an@lpt02 ansible % pwd && ls -la
/usr/local/lib/python3.11/site-packages/ansible_collections/ansible
total 0
drwxr-xr-x   6 an  admin   192 28 Jul 16:02 .
drwxr-xr-x  52 an  admin  1664 28 Jul 16:02 ..
drwxr-xr-x  16 an  admin   512 28 Jul 16:02 netcommon
drwxr-xr-x  17 an  admin   544 28 Jul 16:02 posix
drwxr-xr-x  19 an  admin   608 28 Jul 16:02 utils
drwxr-xr-x  10 an  admin   320 28 Jul 16:02 windows

Also I can't find a way to install the missing module.

The test.yml looks like this:

- hosts: db-server
  become: yes
  become_user: root
  gather_facts: false

  tasks:
    - name: i was here
      file:
        path: /home/bitv/iWasHere.txt
        state: touch

What can I do to solve this problem?

What is the content of your `test.yml` playbook? – β.εηοιτ.βε Jul 31 '23 at 18:27 — β.εηοιτ.βε, Jul 31 '23 at 18:27

score 1 · Accepted Answer · answered Jul 31 '23 at 20:25

I believe the issue is that the ansible.builtin collection is missing even though you have ansible installed. The builtin modules like file, service, etc are now part of ansible.builtin collection.

To fix this, you need to install the ansible collections base package:

pip install ansible-base

This will install the ansible.builtin collection along with other base collections that ansible depends on.

You could also just install the ansible.builtin collection alone:

ansible-galaxy collection install ansible.builtin

After installing the collection, the ModuleNotFoundError should go away when trying to run your playbook.

The other option is to downgrade to ansible 2.9 or earlier, where modules were not packaged into collections. But upgrading and installing the collections would be the recommended solution.

The ansible-galaxy command gives me `ERROR! Failed to resolve the requested dependencies map. Could not satisfy the following requirements` but the installation of ansible-base package fixed it. Thanks! — an28, Jul 31 '23 at 21:23

ModuleNotFoundError: No module named 'ansible_collections.ansible.builtin'

1 Answers1