Accessing / Replacing Yaml values via python using variables

Question

I have been trying to solve what I thought would be simple but can't wrap my head around getting a yaml file updated based on a variable

What I have:

1. An ansible hosts file in YAML format. This hosts file is not 100% the same all the time. It can have a dictionary of multiple image values (as one example) and I only want one to change.

   namespace: demo1
   images:
     image1:
       path: "path1"
       version: "v1"
     image2:
       path: "path2"
       version: "1.2.3"
       user: "root"
   

2. A YAML file that contains the key/values for things I want to replace. We already have a lot of configuration inside this YAML for other parts of our system so I don't want to split off to some other type of config type if I can help it (ini, JSON, etc) I would really want this to be dot notation.

   schema: v1.0
   hostfile:
   - path: path/to/ansible_hosts_file
     images:
       image1.version: v1.1
   

I am trying to find a way to load the YAML from #1, read in the key hostfile.images.[variable] to replace and write back to the original ansible file with the new value. I keep getting tripped up on the variable aspect since today it can be image1.version and the next config its image2.path or both at the same time.

Answer 1

I think your problem primarily comes from mixing key-value pairs with dotted notation. I.e.

images:
  image1.version: v1.1 

instead of doing

images.image1.version: v1.1

Retrieving by dotted notation has been solved for Python and arbitrary separators (not necessarily '.') in this answer. Setting just involves providing two extra functions that take a second argument, which is the value to set and graft them onto CommentedMap resp. CommentesSeq)

Based on that you need to preselect based on your key:

upd = ruamel.yaml.round_trip_load(open('update.yaml')
# schema check here
for hostfile in upd['hostfile']:
   data = ruamel.yaml.round_trip_load(open(hostfile['path']))
   images = data['images']
   for dotted in hostfile['images']:
       val = hostfile['images'][dotted]  
       images.string_set(dotted, val)

The actual string_set-ting code could look like (untested):

def mapping_string_set(self, s, val, delimiter=None, key_delim=None):
    def p(v):
        try:
            v = int(v)
        except:
            pass
        return v
       # possible extend for primitives like float, datetime, booleans, etc.

    if delimiter is None:
        delimiter = '.'
    if key_delim is None:
        key_delim = ','
    try:
        key, rest = s.split(delimiter, 1)
    except ValueError:
        key, rest = s, None
    if key_delim in key:
        key = tuple((p(key) for key in key.split(key_delim)))
    else:
        key = p(key)
    if rest is None:
        self[key] = val
        return 
    self[key].string_set(rest, val, delimiter, key_delim)

ruamel.yaml.comments.CommentedMap.string_set = mapping_string_set

def sequence_string_set(self, s, delimiter=None, key_delim=None):
    if delimiter is None:
        delimiter = '.'
    try:
        key, rest = s.split(delimiter, 1)
    except ValueError:
        key, rest = s, None
    key = int(key)
    if rest is None:
        self[key] = val
        return
    self[key].string_set(rest, val, delimiter, key_delim)

ruamel.yaml.comments.CommentedSeq.string_set = sequence_string_set