Lookup secrets from AWS secret manager | Ansible

Question

Using Terraform code I have created Other type of secrets in AWS Secrets Manager. I need to use these AWS secrets in Ansible code. I found this below link but I am unable to proceed it.

https://docs.ansible.com/ansible/2.8/plugins/lookup/aws_secret.html

I have below Ansible code:-

database.yml

- name: Airflow | DB | Create MySQL DB
  mysql_db:
    login_user: "{{ mysql_user }}"
#    login_password: "{{ mysql_root_password }}"
    login_password: "{{ lookup('ca_dev', 'mysql_root_password') }}"
#    config_file: /etc/my.cnf
#    login_unix_socket: /var/lib/mysql/mysql.sock
#    encrypted: yes
    name: "airflow"
    state: "present"

How can I incorporate AWS secret Manager in my ansible code?

Error message:-

TASK [../../roles/airflow : Airflow | DB | Create MySQL DB] **************************************************************************************************************************************************************************
task path: /home/ec2-user/cng-ansible/roles/airflow/tasks/database.yml:25
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 140, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 539, in _execute
    self._task.post_validate(templar=templar)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 267, in post_validate
    super(Task, self).post_validate(templar)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 364, in post_validate
    value = templar.template(getattr(self, name))
  File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 540, in template
    disable_lookups=disable_lookups,
  File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 495, in template
    disable_lookups=disable_lookups,
  File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 746, in do_template
    res = j2_concat(rf)
  File "<template>", line 8, in root
  File "/usr/lib/python2.7/site-packages/jinja2/runtime.py", line 193, in call
    return __obj(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 631, in _lookup
    instance = self._lookup_loader.get(name.lower(), loader=self._loader, templar=self)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 381, in get
    obj = getattr(self._module_cache[path], self.class_name)
AttributeError: 'module' object has no attribute 'LookupModule'

fatal: [127.0.0.1]: FAILED! => {
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

RUNNING HANDLER [../../roles/airflow : restart rabbitmq-server] 
task path: /home/ec2-user/cng-ansible/roles/airflow/handlers/main.yml:28
    to retry, use: --limit @/home/ec2-user/cng-ansible/plays/airflow/installAirflow.retry

PLAY RECAP
127.0.0.1                  : ok=39   changed=7    unreachable=0    failed=1

ansible-doc -t lookup -l output

I'd remove the quick question at the bottom, it makes the question itself too broad — HermanTheGermanHesse, Apr 24 '19 at 10:28
Can you edit your question and elaborate on "I am unable to proceed it" (what have you tried, what was the result/error msg/log etc....) ? Using aws_secret lookup seems to be exactly what you need. Meanwhile, please note that this module seems to be available only in ansible 2.8 (are you using this version ?) — Zeitounator, Apr 24 '19 at 12:16
@Zeitounator I still did not write any code for this. As Ansible 2.8 release date is May 16th 2019. I don't have any steps to upgrade `ansible 2.7.10` to `2.8` — asur, Apr 24 '19 at 12:51
You can't use a module that is not yet part of your ansible installation. If you need to test in advance, see [how to install a branch/tag from git with pip](https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch). You can either use the `stable-2.8` branch or one of the existing tags (`2.8.0a1` or `2.8.0b1`). [The release is scheduled of may 16th](https://docs.ansible.com/ansible/devel/roadmap/ROADMAP_2_8.html). — Zeitounator, Apr 24 '19 at 13:01
_You can't use a module that is not yet part of your ansible installation._ is not true; you can copy the module from their git repo into the `library` folder of your playbook, and then remove it when the module becomes generally available. I've had to do that several times to backport fixes to AWS modules that they refuse to merge into the 2.7 branch — mdaniel, Apr 24 '19 at 16:04
@MatthewLDaniel I have added `aws_secret.py` under `ansible/modules/cloud/amazon/` path. Updated my ansible script. I am unable to understand how should I access the AWS and lookup for aws secrets — asur, Apr 25 '19 at 12:46
@Zeitounator I have created Ansible code to pull from AWS secrets after adding `aws_secret.py` under `ansible/modules/cloud/amazon/` path. Added error logs in question. — asur, Apr 25 '19 at 12:54

score 1 · Answer 1 · edited Apr 26 '19 at 12:22

1

The error {"msg": "lookup plugin (ca_dev) not found"} suggests your issue is the misuse of the lookup command.

The following line:

login_password: "{{ lookup('ca_dev', 'mysql_root_password') }}"

Should look something like

login_password: "{{ lookup('aws_secret', 'mysql_root_password') }}"

ca_dev is not a valid lookup type, whereas aws_secret is.

You can see a list of supported lookup plugins for Ansible 2.8 in the Lookup Plugins section of the official documentation.

If you are using a custom lookup plugin, or backporting a plugin from a future version of ansible to an older version, you must make sure that it is in a directory visible to ansible.

You can either place the custom file in the default location ansible looks in ~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup or configure your ansible.cfg to look in a different place using the following lookup_plugins ini key under the defaults section.

DEFAULT_LOOKUP_PLUGIN_PATH
Description:    Colon separated paths in which Ansible will search for Lookup Plugins.
Type:   pathspec
Default:    ~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup
Ini Section:    defaults
Ini Key:    lookup_plugins
Environment:    ANSIBLE_LOOKUP_PLUGINS

Documentation for this can be found in the Ansible Configuration section of the official documentation

edited Apr 26 '19 at 12:22

asur

1,759
7
38
81

answered Apr 25 '19 at 13:49

Nick

1,834
20
32

Similar error `TASK [../../roles/airflow : Airflow | DB | Create MySQL DB] fatal: [127.0.0.1]: FAILED! => {"msg": "lookup plugin (aws_secret) not found"}` – asur Apr 25 '19 at 15:01
What is the output of ```ansible --version```?. Alternately do you see aws_secret in the list of available plugins after running ```ansible-doc -t lookup -l``` – Nick Apr 25 '19 at 15:02
I have added aws_secret.py under ansible/modules/cloud/amazon/ path. `https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/aws_secret.py` – asur Apr 25 '19 at 15:05
Do you see it in the list of recognized plugins when running ```ansible-doc -t lookup -l``` – Nick Apr 25 '19 at 15:06
`ansible-doc -t lookup -l` output is added in question. – asur Apr 25 '19 at 15:09
I don't believe ansible is aware of the aws_secret.py file you have added. The default locations ansible looks for this are ```~/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup```. See added comments above. – Nick Apr 25 '19 at 15:12
A different error after adding aws_secrets.py under plugins/look_up. `TASK [../../roles/airflow : Airflow | DB | Create MySQL DB] An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'module' object has no attribute 'LookupModule' fatal: [127.0.0.1]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}` – asur Apr 25 '19 at 15:59
What do you get when you add ```-vvv```? – Nick Apr 25 '19 at 20:45

Lookup secrets from AWS secret manager | Ansible

1 Answers1