Ansible combine "complex" dictionary

Question

Let's say I have the following variables :

services:
  - name: export
    command: "php app/console export"
  - name: import
    command: "php app/console import"

contextes:
  - site_name: mysite
    stages:
      - stage_name: prod
        url: http://mysite.fr
        db:
          name: mysitedb_prod
          user: myuserdb_prod
        # more and more attributes, some simple, some list, some dictionaries
      - stage_name: qualif
        url: http://qualif.mysite.fr
        db:
          name: mysitedb_qualif
          user: mysiteuser_qualif
  - site_name: my2dsite
    stages:
      - stage_name: prod
        url: http://mysite2.fr
        db:
          name: mysite2db_prod
          user: myuser2db_prod

I want to transform them to have the following variable structure :

my_var:
  - site_name: mysite
    stage_name: prod
    services:
      - name: export
        command: "php app/console export"
      - name: import
        command: "php app/console import"
  - site_name: mysite
    stage_name: qualif
    services:
      - name: export
        command: "php app/console export"
      - name: import
        command: "php app/console import"
  - site_name: my2dsite
    stage_name: prod
    services:
      - name: export
        command: "php app/console export"
      - name: import
        command: "php app/console import"

(the final purpose is to create same systemD services for all my sites/stages)

I tried multiple thing based on set_fact and with_subelement loop.

I tought this answer could help me but the problem is not the same and I don't understand the trnasformation made to adapt them. (Note if I repeat services in all site_name/stage_name it work but I would to avoid it)

Note : I can't change the "contextes" variable structure, my all ansible infrastructure is based on.

Thanks a lot

Answer 1

Although I generally agree with @larsk comment on ansible capacity to deal with complex data manipulation, the current problem is not as hard as it seems and can have a fairly concise "ansible only" solution.

My solution relies on the use of a subelements loop

---
- name: Loop on subelements
  hosts: localhost
  gather_facts: false

  vars:
    services:
      - name: export
        command: "php app/console export"
      - name: import
        command: "php app/console import"

    contextes:
      - site_name: mysite
        stages:
          - stage_name: prod
            url: http://mysite.fr
            db:
              name: mysitedb_prod
              user: myuserdb_prod
            # more and more attributes, some simple, some list, some dictionaries
          - stage_name: qualif
            url: http://qualif.mysite.fr
            db:
              name: mysitedb_qualif
              user: mysiteuser_qualif
      - site_name: my2dsite
        stages:
          - stage_name: prod
            url: http://mysite2.fr
            db:
              name: mysite2db_prod
              user: myuser2db_prod

  tasks:
    - name: Construct my new data structure
      vars:
        current_hash:
          site_name: "{{ item.0.site_name }}"
          stage_name: "{{ item.1.stage_name }}"
          services: "{{ services }}"
      set_fact:
        my_var: "{{ my_var | default([]) + [current_hash] }}"
      loop: "{{ lookup('subelements', contextes, 'stages') }}"

    - name: Show calculated var
      debug:
        var: my_var

Which gives

$ ansible-playbook test.yml 

PLAY [Loop on subelements] **********************************************************************************************************************************************************************************************************************************************

TASK [Construct my new data structure] **********************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=[{'site_name': 'mysite'}, {'stage_name': 'prod', 'url': 'http://mysite.fr', 'db': {'name': 'mysitedb_prod', 'user': 'myuserdb_prod'}}])
ok: [localhost] => (item=[{'site_name': 'mysite'}, {'stage_name': 'qualif', 'url': 'http://qualif.mysite.fr', 'db': {'name': 'mysitedb_qualif', 'user': 'mysiteuser_qualif'}}])
ok: [localhost] => (item=[{'site_name': 'my2dsite'}, {'stage_name': 'prod', 'url': 'http://mysite2.fr', 'db': {'name': 'mysite2db_prod', 'user': 'myuser2db_prod'}}])

TASK [Show calculated var] **********************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "my_var": [
        {
            "services": [
                {
                    "command": "php app/console export",
                    "name": "export"
                },
                {
                    "command": "php app/console import",
                    "name": "import"
                }
            ],
            "site_name": "mysite",
            "stage_name": "prod"
        },
        {
            "services": [
                {
                    "command": "php app/console export",
                    "name": "export"
                },
                {
                    "command": "php app/console import",
                    "name": "import"
                }
            ],
            "site_name": "mysite",
            "stage_name": "qualif"
        },
        {
            "services": [
                {
                    "command": "php app/console export",
                    "name": "export"
                },
                {
                    "command": "php app/console import",
                    "name": "import"
                }
            ],
            "site_name": "my2dsite",
            "stage_name": "prod"
        }
    ]
}

PLAY RECAP **************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0