Working with ansible's variables, lists and dicts

Question

I want to deploy sensitive data using ansible-vault. Basically I want to iterate through every repository, iterate through its config files looking for secrets kept in the vault and replace them with the encrypted value in the vault. I want to pass the values of repositories as a variable to look through the secrets but keep getting errors.

I tried various approaches but I'm unsure how to work with these data structures.
Advice is very appreciated.

My code structure:

vars:
  repositories: ["service1", "service2"]
  service1:
    service1.conf: ["vault_user1", "vault_password1"]
    service1.env: ["vault_secret1"]
  service2:
    service2.conf: ["vault_user2", "vault_password2"]

- name: place secrets
  replace:
    path: "/tmp/{{ repositories }}/{{ item.key }}"
    regexp: "{{ item.value }}"
    replace: "{{ item.value }}"
  with_items: "{{ repositories }} | dict2items"

vault.yaml

vault_user1: a
vault_password1: b
vault_user2: c
vault_password1: d
vault_secret1: e

service1.conf

user=vault_user1
password=vault_password1
secret=vault_secret1

service2.conf

user=vault_user2
password=vault_password2

why do you want to do that in such a difficult way? You can use templates for your configuration files. — toydarian, Oct 05 '20 at 07:01
Your `replace` operation is searching for the exact same thing it replaces. Is that a copy/paste error ? Beside that, as pointed out by @toydarian you can simply use templates. — Zeitounator, Oct 05 '20 at 09:07
I would like to keep as much information as possible in the repositories of the services. If I start templating, the config files move to the ansible repository and I don't like it being split like this. Yes the replace operation is nonsense currently. Basically regexp is supposed to look for 'vault_user1' and replace it with 'a'. — Granolaboy, Oct 05 '20 at 10:38

Vladimir Botka · Accepted Answer · 2020-10-05T16:16:43.907

Given the file with the variables (encrypted or not doesn't make any difference in this example) and the files (rooted in the local directory tmp in this example).

shell> cat vault.yaml
vault_user1: a
vault_password1: b
vault_user2: c
vault_password2: d
vault_secret1: e

shell> tree tmp
tmp
├── service1
│   ├── service1.conf
│   └── service1.env
└── service2
    └── service2.conf

shell> cat tmp/service1/service1.conf 
vault_user1
vault_password1
vault_secret1
vault_user2
vault_password2

shell> cat tmp/service1/service1.env 
vault_user1
vault_password1
vault_secret1
vault_user2
vault_password2

shell> cat tmp/service2/service2.conf 
vault_user1
vault_password1
vault_secret1
vault_user2
vault_password2

To solve the problem, there must be 2 nested loops. Iterate repositories in the first loop and a list of the variables in the second. This can be achieved by iterating include_tasks with the second loop inside the file. For example, create a file

shell> cat place_secrets.yml
- name: place secrets
  replace:
     path: "tmp/{{ dir }}/{{ item.0.key }}"
     regexp: "{{ item.1 }}"
     replace: "{{ my_vault[item.1] }}"
  with_subelements:
    - "{{ repo|dict2items }}"
    - value

In the playbook, read the variables into the dictionary my_vault, and iterate the list repositories

- hosts: localhost
  vars:
    repositories: [service1,service2]
    service1:
      service1.conf: [vault_user1, vault_password1]
      service1.env: [vault_secret1]
    service2:
      service2.conf: [vault_user2, vault_password2]

  tasks:
    - include_vars:
        file: vault.yaml
        name: my_vault
    - include_tasks: place_secrets.yml
      loop: "{{ repositories }}"
      loop_control:
        loop_var: outer_item
      vars:
        dir: "{{ outer_item }}"
        repo: "{{ lookup('vars', outer_item) }}"

gives

shell> cat tmp/service1/service1.conf 
a
b
vault_secret1
vault_user2
vault_password2

shell> cat tmp/service1/service1.env 
vault_user1
vault_password1
e
vault_user2
vault_password2

shell> cat tmp/service2/service2.conf 
vault_user1
vault_password1
vault_secret1
c
d

If the structure of the data can be simplified the playbook below gives the same results

- hosts: localhost
  vars:
    repositories:
      service1.conf: [vault_user1, vault_password1]
      service1.env: [vault_secret1]
      service2.conf: [vault_user2, vault_password2]

  tasks:
    - include_vars:
        file: vault.yaml
        name: my_vault
    - name: place secrets
      replace:
         path: "tmp/{{ (item.0.key|splitext).0 }}/{{ item.0.key }}"
         regexp: "{{ item.1 }}"
         replace: "{{ my_vault[item.1] }}"
      with_subelements:
         - "{{ repositories|dict2items }}"
         - value

Thank you for your thorough answer, I think I can apply the simplified version using inventory variables inside AWX. I'll get back to you in case I have more questions. Again, big thanks!! Is there a reason you don't use quotation marks in the dicts? — Granolaboy, Oct 05 '20 at 13:23
See [Do I need quotes for strings in YAML?](https://stackoverflow.com/questions/19109912/do-i-need-quotes-for-strings-in-yaml#:~:text=In%20general%2C%20you%20don't%20need%20quotes.,%25%20%2C%20%40%20%2C%20%5C%20). — Vladimir Botka, Oct 05 '20 at 15:39

Working with ansible's variables, lists and dicts

1 Answers1