What is the best way of modifying and replacing list elements?

Question

The original list has the following structure:

endpoints:
-   address: 10.10.10.1
    name: hostname1:867
    write: yes
-   address: 10.10.10.2
    name: hostname2:867
    write: yes
-   address: 10.10.10.3
    name: hostname3:867
    write: yes

I'm trying to create a new list by splitting the "name" field, resulting in something like this:

endpoints:
-   address: 10.10.10.1
    name: hostname1
    port: 867
    write: yes
-   address: 10.10.10.2
    name: hostname2
    port: 867
    write: yes
-   address: 10.10.10.3
    name: hostname3
    port: 867
    write: yes

I've tried a combination of json_query and map('regex_replace') and then setting a new fact but wasn't successful.

"I've tried a combination of json_query and map('regex_replace') and then setting a new fact but wasn't successful." - Showing us the code rather than the description of the code would be much more useful. — Amadan, Jul 30 '19 at 04:02

score 0 · Answer 1 · answered Jul 30 '19 at 05:37

While it seems like the logical choice to use json_query, I'm not sure how it would work since JMESPath (the project json_query is built on) lacks a split function.

You could consider writing your own filter plugin. It adds complexity for future playbook maintenance, but data manipulation in this way is so much easier in python.

In your playbook directory, create a folder called filter_plugins and a file split_hostname.py with contents:

#!/usr/bin/env python

class FilterModule(object):
    def filters(self):
        return {'split_hostname': split_hostname}

def split_hostname(endpoints):
    new_endpoints = []
    for endpoint in endpoints:
        hostname, port = endpoint["name"].split(":")
        endpoint["name"] = hostname
        endpoint["port"] = port
        new_endpoints.append(endpoint)
    return new_endpoints

Then use the filter in your playbook:

    - set_fact:
        new_endpoints: "{{ endpoints | split_hostname }}"

This is a very basic example. If you choose to go down this path, you may want to add some error handling.

I was trying to avoid having to write my own if I could use what was already available. Thanks for your suggestion I will definitely keep this in mind for the future. It is similar to this idea - https://stackoverflow.com/questions/46025695/best-way-to-modify-json-in-ansible — honeybeeham, Jul 30 '19 at 13:29

Vladimir Botka · Accepted Answer · 2019-08-06T01:54:33.983

The tasks below

- set_fact:
    ep2: "{{ ep2|
             default([]) + [
             {'write': item.write,
             'address': item.address,
             'port': item.name.split(':').1,
             'name': item.name.split(':').0} ]
             }}"
  loop: "{{ endpoints }}"
- debug:
    var: ep2

give

  ep2:
  - address: 10.10.10.1
    name: hostname1
    port: '867'
    write: true
  - address: 10.10.10.2
    name: hostname2
    port: '867'
    write: true
  - address: 10.10.10.3
    name: hostname3
    port: '867'
    write: true

Task below is more flexible and adds items write and address only if defined

- set_fact:
    ep: "{{ ep|
            default([]) + [ {}|
            combine((item.write is defined)|
                     ternary({'write': item.write}, {}))|
            combine((item.address is defined)|
                     ternary({'address': item.address}, {}))|
            combine({'port': item.name.split(':').1})|
            combine({'name': item.name.split(':').0}) ]
            }}"
  loop: "{{ endpoints }}"

If the complete item can be used then the task can be simplified

- set_fact:
    ep: "{{ ep|
            default([]) + [
            item|
            combine({'port': item.name.split(':').1})|
            combine({'name': item.name.split(':').0}) ]
            }}"
  loop: "{{ endpoints }}"

If you want to try filter_plugins use dict_utils. Below is the simplified dict_add_hash filter.

$ cat filter_plugins/dict_utils.py
def dict_add_hash(d, h):
    for k, v in h.iteritems():
        d[k] = v
    return d

class FilterModule(object):
    ''' Ansible filters. Interface to Python dictionary methods.'''

    def filters(self):
        return {
            'dict_add_hash' : dict_add_hash
        }

the tasks

- set_fact:
    ep2: "{{ ep2|
             default([]) + [
             item|
             dict_add_hash({'port': item.name.split(':').1})|
             dict_add_hash({'name': item.name.split(':').0}) ]
             }}"
  loop: "{{ endpoints }}"
- debug:
    var: ep2

give the same result

  ep2:
  - address: 10.10.10.1
    name: hostname1
    port: '867'
    write: true
  - address: 10.10.10.2
    name: hostname2
    port: '867'
    write: true
  - address: 10.10.10.3
    name: hostname3
    port: '867'
    write: true

See all available plugins.

The first method is what I was looking for! But I can see the dict_utils being handy in cases with larger data sets. Thanks! — honeybeeham, Jul 30 '19 at 13:35
Using the first method, is there a way to apply filters to omit a field in the newly defined dict? Example, something like `'write': item.write, | default(omit)` — honeybeeham, Aug 05 '19 at 22:30
Sure. Use filters *combine* and *ternary*. I've updated the answer with an example. — Vladimir Botka, Aug 06 '19 at 01:35

What is the best way of modifying and replacing list elements?

2 Answers2