best way to modify json in ansible

Question

I have a variable (via set_fact) containing a json string:

{
  "PolicyVersion": {
    "CreateDate": "2017-08-07T02:48:05Z",
    "Document": {
      "Statement": [
        {
          "Action": "sts:AssumeRole",
          "Effect": "Allow",
          "Resource": [
            "arn:aws:iam::123456789123:role/Root_Update_svcacct",
            "arn:aws:iam::123456789123:role/Root_Delete_svcacct",
            "arn:aws:iam::123456789123:role/Root_Create_svcacct",
            "arn:aws:iam::123456789123:role/Root_Full_svcacct",
            "arn:aws:iam::987654321987:role/Member1_Create_svcacct",
            "arn:aws:iam::987654321987:role/Member1_Update_svcacct",
            "arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
            "arn:aws:iam::987654321987:role/Member1_Full_svcacct"
          ]
        }
      ],
      "Version": "2012-10-17"
    },
    "IsDefaultVersion": true,
    "VersionId": "v2"
  }
}

What is the best way to insert more elements in the "Resource" array?

"arn:aws:iam::001122334455:role/Member1_Create_svcacct",
"arn:aws:iam::001122334455:role/Member1_Update_svcacct",
"arn:aws:iam::001122334455:role/Member1_Delete_svcacct",
"arn:aws:iam::001122334455:role/Member1_Full_svcacct"

I am exploring dumping the variable to a file and inserting the block I want with external shell tools, which does not seem to be elegant.

score 12 · Accepted Answer · answered Sep 04 '17 at 02:03

I don't know about the best way, but one option is to write a simple library module to handle the mechanics of the update for you. You could use the jsonpointer module as a way of locating the data you wish to modify, and then return the modified object to ansible. A starting point might look like:

#!/usr/bin/python

from ansible.module_utils.basic import AnsibleModule

import json

try:
    import jsonpointer
except ImportError:
    jsonpointer = None


def main():
    module = AnsibleModule(
        argument_spec=dict(
            data=dict(required=True, type='dict'),
            pointer=dict(required=True),
            action=dict(required=True,
                        choices=['append', 'extend', 'update']),
            update=dict(type='dict'),
            extend=dict(type='list'),
            append=dict(),
        ),
        supports_check_mode=True,
    )

    if jsonpointer is None:
        module.fail_json(msg='jsonpointer module is not available')

    action = module.params['action']
    data = module.params['data']
    pointer = module.params['pointer']

    if isinstance(data, str):
        data = json.loads(str)

    try:
        res = jsonpointer.resolve_pointer(data, pointer)
    except jsonpointer.JsonPointerException as err:
        module.fail_json(msg=str(err))

    if action == 'append':
        res.append(module.params['append'])
    if action == 'extend':
        res.extend(module.params['extend'])
    elif action == 'update':
        res.update(module.params['update'])

    module.exit_json(changed=True,
                     result=data)


if __name__ == '__main__':
    main()

If you drop this into, e.g., library/json_modify.py, you can use it in a playbook like this:

- hosts: localhost
  gather_facts: false
  vars:
    myvar: {
        "PolicyVersion": {
          "CreateDate": "2017-08-07T02:48:05Z",
          "Document": {
            "Statement": [
              {
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Resource": [
                  "arn:aws:iam::123456789123:role/Root_Update_svcacct",
                  "arn:aws:iam::123456789123:role/Root_Delete_svcacct",
                  "arn:aws:iam::123456789123:role/Root_Create_svcacct",
                  "arn:aws:iam::123456789123:role/Root_Full_svcacct",
                  "arn:aws:iam::987654321987:role/Member1_Create_svcacct",
                  "arn:aws:iam::987654321987:role/Member1_Update_svcacct",
                  "arn:aws:iam::987654321987:role/Member1_Delete_svcacct",
                  "arn:aws:iam::987654321987:role/Member1_Full_svcacct"
                ]
              }
            ],
            "Version": "2012-10-17"
          },
          "IsDefaultVersion": true,
          "VersionId": "v2"
        }
      }
  tasks:
    - json_modify:
        data: "{{ myvar }}"
        pointer: "/PolicyVersion/Document/Statement/0/Resource"
        action: extend
        extend:
          - "arn:aws:iam::001122334455:role/Member1_Create_svcacct"
          - "arn:aws:iam::001122334455:role/Member1_Update_svcacct"
          - "arn:aws:iam::001122334455:role/Member1_Delete_svcacct"
          - "arn:aws:iam::001122334455:role/Member1_Full_svcacct"
      register: result

    - debug:
        var: result.result

The result of running this playbook and the proposed module is:

TASK [debug] *******************************************************************
ok: [localhost] => {
    "result.result": {
        "PolicyVersion": {
            "CreateDate": "2017-08-07T02:48:05Z", 
            "Document": {
                "Statement": [
                    {
                        "Action": "sts:AssumeRole", 
                        "Effect": "Allow", 
                        "Resource": [
                            "arn:aws:iam::123456789123:role/Root_Update_svcacct", 
                            "arn:aws:iam::123456789123:role/Root_Delete_svcacct", 
                            "arn:aws:iam::123456789123:role/Root_Create_svcacct", 
                            "arn:aws:iam::123456789123:role/Root_Full_svcacct", 
                            "arn:aws:iam::987654321987:role/Member1_Create_svcacct", 
                            "arn:aws:iam::987654321987:role/Member1_Update_svcacct", 
                            "arn:aws:iam::987654321987:role/Member1_Delete_svcacct", 
                            "arn:aws:iam::987654321987:role/Member1_Full_svcacct", 
                            "arn:aws:iam::001122334455:role/Member1_Create_svcacct", 
                            "arn:aws:iam::001122334455:role/Member1_Update_svcacct", 
                            "arn:aws:iam::001122334455:role/Member1_Delete_svcacct", 
                            "arn:aws:iam::001122334455:role/Member1_Full_svcacct"
                        ]
                    }
                ], 
                "Version": "2012-10-17"
            }, 
            "IsDefaultVersion": true, 
            "VersionId": "v2"
        }
    }
}

Ansible will look in the library directory relative to the playbook. In this example, put the custom module in `playbook/library/json_modify.py`. — Sion, Jul 04 '18 at 06:43
@larsks I realize this thread is old, but the code for json_modify appears to change the resulting json to add an extra closing bracket at the end. The result fails. — Juan Jimenez, Oct 10 '18 at 14:21
@JuanJimenez I don't think it does: the code doesn't do anything that could add an unbalanced bracket like that (it only manipulates the data using the `json` and `jsonpointer` modules, both of which are pretty stable). If you have a link to a failing example, let me know and I would be happy to update the answer. — larsks, Oct 10 '18 at 14:43
Thanks, it works great. Sad to see there is no Ansible way to do that instead. I played around with set_fact and combine, but no luck. Better to do it in pure python via a module... but sad!! — fabiog1901, Nov 25 '19 at 21:36

score 2 · Answer 2 · answered May 27 '20 at 12:52

2

Actually Ansible is natively able to read JSON files.

see this question:

add-a-new-key-value-to-a-json-file-using-ansible

answered May 27 '20 at 12:52

Emmanuel

177
3

score 0 · Answer 3 · edited May 05 '21 at 06:56

This is a little bit old. I know but I felt it could be useful to many people so I'll post here my solution as "the best way to modify a json" that I could personally find.

First of all the JSON. In my use-case I had a bunch of aws snapshots that were encrypted with the wrong kms key and I had to recreate the AMI with the correct key.

So I had to :

get data of the old snapshots ( like size device_name etc.)
create the new snaps with the different key
re-register a new ami with the correct block_device_mapping Here's the code

      - name: get ami
        amazon.aws.ec2_ami_info:
          image_ids: ami-<id>
          region: "{{ region }}"
        register: ami
      
      - name: save snapshot ids and device_name and volume_size
        set_fact:
          snapshot_ids: "{{ ami | json_query('images[].block_device_mappings[].ebs.snapshot_id') }}"
          device_name: "{{ ami  | json_query('images[].block_device_mappings[].device_name') }}"
          volume_size: "{{ ami | json_query('images[].block_device_mappings[].ebs.volume_size') }}"

basically each of the above is a list of each of the 3 things (device_name, snap_id, volume_size) that I need (but it could be extended)

then:

      - name: get kms arn
        aws_kms_info:
          filters:
            alias: "{{ kms_keys.alias }}"
          region: "{{ region }}"
        register: aws_kms_facts_out
      - debug:
          var: aws_kms_facts_out
      - set_fact:
          kms_arn: "{{ aws_kms_facts_out['keys'][0].key_arn }}"

      - name: copy snap with new encryption key
        community.aws.ec2_snapshot_copy:
          source_region: "{{ region }}"
          region: "{{ region }}"
          source_snapshot_id: "{{ item }}"
          encrypted: yes
          kms_key_id: "{{ kms_arn }}"
        loop: "{{ snapshot_ids }}"
        register: new_snapshots

and then here's the catch

      - set_fact:
          new_snapshot_ids: "{{ new_snapshots| json_query('snapshots[].snapshot_id') }}"

      - name: creating the block_device_mappings structure (still a list of objects)
        ansible.builtin.debug:
          msg: '{
                  "device_name": "{{ item.2 }}",
                  "delete_on_termination": "true",
                  "snapshot_id": "{{ item.0 }}",
                  "volume_size": {{ item.1 }},
                  "volume_type": "gp2"
                }'
        loop: "{{ new_snapshot_ids|zip_longest(volume_size,device_name)|list }}"
        register: block_device_mappings  

      - set_fact:
          bdm: "{{ block_device_mappings | json_query('results[].msg') }}"

finally

      - name: create new ami from newly created snapshots
        amazon.aws.ec2_ami:
          region: "{{ region }}"
          name: "{{ instance.name }}-{{ ansible_date_time.date }}"
          state: present
          architecture: x86_64
          virtualization_type: hvm
          root_device_name: /dev/sda1
          device_mapping: "{{ bdm }}"

This is how you can do it without requiring any additional trick. Of course this is declined to my particular use case but you can adapt it to any circumstance, that do not require a complete disassemble and reassemble of the Json itself

best way to modify json in ansible

3 Answers3

Linked