How to include variables with include_vars with the same name without overwriting previous

Question

I am having this let's call it include.yaml

#- name: "Playing with Ansible and Include files"
- hosts: localhost
  connection: local 
  tasks:
    - find: paths="./" recurse=yes patterns="test.yaml"
      register: file_to_exclude
    - debug: var=file_to_exclude.stdout_lines        
    - name: shell
      shell: "find \"$(pwd)\" -name 'test.yaml'"
      register: files_from_dirs     
    - debug: var=files_from_dirs.stdout_lines
    - name: Include variable files
      include_vars: "{{ item }}"
      with_items:
        - "{{ files_from_dirs.stdout_lines }}"
    - debug: var=files

and 2 ore more test files

./dir1/test.yaml

that contains

files: 
  - file1 
  - file2

./dir2/test.yaml

that contains

files: 
  - file3 
  - file4

the result is

TASK [Include variable files] ******************************************************************************************
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/1st/test.yaml)
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/2nd/test.yaml)

TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
    "files": [
        "file3",
        "file4"
    ]
}

How can I get all the values in files, at the moment the last included files variable from last file overrides the files from the previous files? Of course without changing the variables names in files test.yaml?

In other words I want files to be:

ok: [localhost] => {
    "files": [
        "file1",
        "file2",
        "file3",
        "file4"
    ]
}

To be more specific, I ask for any kind of solution or module, even not official or some github module, I don't want a specific include_vars module solution.

Since you are like to append, concatenate or combine variables sourced from different var files, as well overriding the variable precedence behavior, you'll probably find the answer under [In Ansible, how to combine variables from separate files into one array?](https://stackoverflow.com/questions/35554415/in-ansible-how-to-combine-variables-from-separate-files-into-one-array). — U880D, Sep 02 '22 at 11:26
@U880D I think the question you directed in the comment has definitely similar subject, but the question I put is very specific, and since I ask for a solution without specifying exactly which modules to use, also the answers in the link you directed are more than 6 years and possibly obsolete and many of them are contradictory or low quality... — Eduard Florinescu, Sep 02 '22 at 12:02
Even if your question is specific, the root cause of what you are observing is [variable precedence](https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable) and hasn't changed. Ansible still behaves in the same way. You need either write your own vars plugin, change your data structure, merge the files before including or [change how Ansible behaves (but that seems to be deprecated now)](https://github.com/ansible/ansible/issues/73089). — U880D, Sep 02 '22 at 12:10
@U880D if you check the response and git repo solution the response is related to `ansible` version `1.x` — Eduard Florinescu, Sep 02 '22 at 12:17
Maybe [`update_fact` module – Update currently set facts](https://docs.ansible.com/ansible/latest/collections/ansible/utils/update_fact_module.html) can provide some advantage in your case? — U880D, Sep 02 '22 at 12:17
@U880D I will check, also I think this might also help https://stackoverflow.com/questions/71928265/how-to-loop-over-same-keys-in-different-yaml-files-to-generate-an-output-per-key — Eduard Florinescu, Sep 02 '22 at 12:19
Right, but even if the answers are old, the root cause of what you are observing is variable precedence, hasn't changed and is exactly documented as you are observing. — U880D, Sep 02 '22 at 12:19
@U880D that happens with `include_vars`, but I haven't tied the solution to `include_vars` so maybe there are workarounds — Eduard Florinescu, Sep 02 '22 at 12:20
Yes, there are workarounds since you need only to add additional elements to a list. So your question is just about "_How to append items to an existing list?_". — U880D, Sep 02 '22 at 12:26

Vladimir Botka · Accepted Answer · 2022-09-06T16:18:44.203

Put the included variables into the dictionaries with unique names. For example, create the names from the index of the loop. Then, iterate the names and concatenate the lists

    - command: "find {{ playbook_dir }} -name test.yaml"
      register: files_from_dirs

    - include_vars:
        file: "{{ item }}"
        name: "{{ name }}"
      loop: "{{ files_from_dirs.stdout_lines }}"
      loop_control:
        extended: true
      vars:
        name: "files_{{ ansible_loop.index }}"

    - set_fact:
        files: "{{ files|d([]) + lookup('vars', item).files }}"
      with_varnames: "files_[0-9]+"

    - debug:
        var: files

give

files:
  - file1
  - file2
  - file3
  - file4

^{Notes:

You have to provide either a path relative to the home directory or an absolute path. See the example below

- command: "echo $PWD"
register: out
- debug:
var: out.stdout

give
out.stdout: /home/admin

For example, when you want to find the files relative to the directory of the playbook
- command: "find {{ playbook_dir }} -name test.yaml"
register: files_from_dirs
- debug:
var: files_from_dirs.stdout_lines

give
files_from_dirs.stdout_lines:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml

The same is valid for the module find. For example,

- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs
- debug:
var: files_from_dirs.files|map(attribute='path')|list

give the same result
files_from_dirs.files|map(attribute='path')|list:
- /export/scratch/tmp8/test-987/dir1/test.yaml
- /export/scratch/tmp8/test-987/dir2/test.yaml

Simplify the code and put the declaration of files into the vars. For example, the below declaration gives the same result

files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"

Example of a complete playbook for testing
- hosts: localhost

vars:

files: "{{ query('varnames', 'files_[0-9]+')|
map('extract', hostvars.localhost, 'files')|
flatten }}"

tasks:

- find:
paths: "{{ playbook_dir }}"
recurse: true
patterns: test.yaml
register: files_from_dirs

- include_vars:
file: "{{ item }}"
name: "{{ name }}"
loop: "{{ files_from_dirs.files|map(attribute='path')|list }}"
loop_control:
extended: true
vars:
name: "files_{{ ansible_loop.index }}"

- debug:
var: files

(maybe off-topic, see comments)
Q: "Is there a way to write the path where it was found?"
A: Yes, it is. See the self-explaining example below. Given the inventory
shell> cat hosts
host_1 file_1=alice
host_2 file_2=bob
host_3

the playbook
- hosts: host_1,host_2,host_3

vars:

file_1_list: "{{ hostvars|json_query('*.file_1') }}"
file_2_list: "{{ hostvars|json_query('*.file_2') }}"
file_1_dict: "{{ dict(hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].[key, value.file_1]')) }}"
file_1_lis2: "{{ hostvars|dict2items|
selectattr('value.file_1', 'defined')|
json_query('[].{key: key, file_1: value.file_1}') }}"

tasks:

- debug:
msg: |-
file_1_list: {{ file_1_list }}
file_2_list: {{ file_2_list }}
file_1_dict:
{{ file_1_dict|to_nice_yaml|indent(2) }}
file_1_lis2:
{{ file_1_lis2|to_nice_yaml|indent(2) }}
run_once: true

gives
msg: |-
file_1_list: ['alice']
file_2_list: ['bob']
file_1_dict:
host_1: alice

file_1_lis2:
- file_1: alice
key: host_1}

is there an alternative to taking them from `hostvars.localhost`? — Eduard Florinescu, Sep 06 '22 at 11:30
Sure. Generally, use `hostvars[inventory_hostname]` . In the above example `hostvars.localhost` is a special case where `inventory_hostname` is `localhost`. You can use *hostvars* of other hosts too, depending on the use case, of course. Open a new question if you have trouble with another use case. — Vladimir Botka, Sep 06 '22 at 11:37
the variables are definitely on the control machine, since I am not running in ansbile but zuul, but I don't know if zuul changes the place where it keeps those ansible variables, I don't know where to search them either, and I am thinking maybe they are in another place, the code works on my local but on zuul is says Key not found `files_1`, etc — Eduard Florinescu, Sep 06 '22 at 11:42
is there a way to search for the `files_1` key in the whole variables space recursively — Eduard Florinescu, Sep 06 '22 at 11:44
Sure. Use `{{ hostvars | dict2items | json_query('[].value.files_1') }}` — Vladimir Botka, Sep 06 '22 at 11:47
I am not finding it in `hostvars` I will try `job.host-vars`, but isn't a way to pin those `files_1` under a specific variable I define? — Eduard Florinescu, Sep 06 '22 at 13:07
also is there a way to write the whole dictionary structure, with keys and objects inside? — Eduard Florinescu, Sep 06 '22 at 13:09
Of course, it is. e.g. `{{ hostvars | dict2items | json_query('[].{k1: value.files_1, k2: value.files_2') }}` **Open a new question!** — Vladimir Botka, Sep 06 '22 at 13:31
https://stackoverflow.com/questions/73623257/how-can-i-find-a-key-deep-inside-in-nested-dictionaries-and-show-the-path-to-tha — Eduard Florinescu, Sep 06 '22 at 13:54
I cannot find nothing in `hostvars` either `localhost` or other `host` I think this might be a feature or characteristic of `zuul-ci` I put another question with my next issue https://stackoverflow.com/questions/73635936/hostvar-in-playbook-is-not-accessible-in-zuul-while-it-works-in-ansible — Eduard Florinescu, Sep 07 '22 at 12:59

How to include variables with include_vars with the same name without overwriting previous

1 Answers1