Update YAML file programmatically

Question

I've a Python dict that comes from reading a YAML file with the usual

yaml.load(stream)

I'd like to update the YAML file programmatically given a path to be updated like:

group1,option1,option11,value

and save the resulting dict again as a yaml file. I'm facing the problem of updating a dicntionary, taking into account that the path is dynamic (let's say a user is able to enter the path through a simple CLI I've created using Cmd).

Any ideas?

thanks!

UPDATE Let me be more specific on the question: The issue is with updating part of a dictionary where I do not know in advance the structure. I'm working on a project where all the configuration is stored on YAML files, and I want to add a CLI to avoid having to edit them by hand. This a sample YAML file, loaded to a dictionary (config-dict) using PyYaml:

config:
 a-function: enable
 b-function: disable
 firewall:
  NET:
   A:
    uplink: enable
    downlink: enable
   B:
    uplink: enable
    downlink: enable
  subscriber-filter:
   cancellation-timer: 180
 service:
  copy:
   DS: enable
  remark:
   header-remark:
    DSC: enable
    remark-table:
 port:
  linkup-debounce: 300
  p0:
   mode: amode
  p1:
   mode: bmode
  p2:
   mode: amode
  p3:
   mode: bmode

I've created the CLI with Cmd, and it's working great even with autocompletion. The user may provide a line like:

config port p1 mode amode

So, I need to edit:

config-dict['config']['port']['p1']['mode'] and set it to 'amode'. Then, use yaml.dump() to create the file again. Another possible line would be:

config a-function enable

So config-dict['config']['a-function'] has to be set to 'enable'.

My problem is when updating the dictionary. If Python passed values as a reference would be easy: Just iterate through the dict until the right value is found and save it. Actually this is what I'm doing for the Cmd autocomplete. But I don't know how to do the update.

Hope I explained myself better now!

Thanks in advance.

Saving to YAML is `yaml.dump` easy, but you are asking for design of your app. This is too difficult, opinion based, and in fact does not have much to do with YAML. — Jan Vlcinsky, May 28 '14 at 14:27
@JanVlcinsky thanks for your answer. You are right, I'm asking more about Python than YAML itself. It's not about the design but a very specific question: How to update a dictionary which has a dynamic size. — Ignacio Verona, May 28 '14 at 14:35
Euh, I don't see an issue here - really straight-forward way would be to parse the path given on the cli, walk through your dictionary to ensure path is valid, change the value, dump. Where's the issue? Other than that - what have you tried, what didn't work, paste some code. — favoretti, May 28 '14 at 14:51
Possible duplicate of [Access python nested dictionary items via a list of keys](http://stackoverflow.com/questions/14692690/access-python-nested-dictionary-items-via-a-list-of-keys) — Brian Tompsett - 汤莱恩, Jun 13 '16 at 16:46

score 21 · Answer 1 · answered May 28 '14 at 15:04

21

In fact the solution follows simple patter: load - modify - dump:

Before playing, be sure you have pyyaml installed:

$ pip install pyyaml

`testyaml.py`

import yaml
fname = "data.yaml"

dct = {"Jan": {"score": 3, "city": "Karvina"}, "David": {"score": 33, "city": "Brno"}}

with open(fname, "w") as f:
    yaml.dump(dct, f)

with open(fname) as f:
    newdct = yaml.load(f)

print newdct
newdct["Pipi"] = {"score": 1000000, "city": "Stockholm"}

with open(fname, "w") as f:
    yaml.dump(newdct, f)

Resulting `data.yaml`

$ cat data.yaml
David: {city: Brno, score: 33}
Jan: {city: Karvina, score: 3}
Pipi: {city: Stockholm, score: 1000000}

answered May 28 '14 at 15:04

Jan Vlcinsky

42,725
12
101
98

Thanks @Jan. The problem here is that the update path is not fixed. I might be updaing dict['A']['a1']['a11'] or dict['A']['a2']['a21']['a221'], depending on the input from the user. So, I'm trying to figure out how to loop through the dict based on a "key path" (something like 'A,a1,a11' and update that value. – Ignacio Verona May 28 '14 at 15:21
@IgnacioVerona For modifying dict with dynamic number of keys see http://stackoverflow.com/questions/17462011/python-generate-a-dynamic-dictionary-from-the-list-of-keys – Jan Vlcinsky May 28 '14 at 15:39
that works for inserting, but what about updating? If python allowed pass-by-reference this will be easy, but as it does not it's a bit challenging for me! – Ignacio Verona May 28 '14 at 15:49

score 4 · Answer 2 · edited Oct 29 '20 at 17:20

It is very simple to do it with python-benedict, a solid python dict subclass that support IO operations with many formats, including yaml.

Installation: pip install python-benedict

You can initialize it directly from the yaml file:

from benedict import benedict

f = 'data.yaml'
d = benedict.from_yaml(f)
d['Pipi'] = {'score': 1000000, 'city': 'Stockholm'}

# benedict supports keypath (dot syntax by default),
# so it's possible to update nested values easily:
d['Pipi.score'] = 2000000
print(d['Pipi']) # -> {'score': 2000000, 'city': 'Stockholm'}

d.to_yaml(filepath=f)

Here the library repository and the documentation: https://github.com/fabiocaccamo/python-benedict

Note: I am the author of this project

Dan Temkin · Answer 3 · 2018-01-02T23:50:44.197

Updating seems to the one place where pyyaml falls short. You cannot even use yaml.load on a file that was opened in (a)ppend mode without an exception. Now this may be a bit tedious for complex dictionaries but if each added item represents a separate case or document you could handle it as if it were any other text file.

newinfo = {"Pipi": {"score": 100000, "city": "Stockholm"}}
with open(fname, "a") as f:
     sep = "\n" # For distinct documents use "\n...\n" as separator

     # Pay attention to where you put the separator. 
     # if the file exists and is in traditional format place at 
     # beginning of string. else place at the end.

     infostring = "{}".format(newinfo)
     f.write(infostring + sep)

While this doesn't necessarily help with value updating, it does allow for file updating. You might also look into using json.dump on the file. I know it is in YAML but the formats are largely compatible unless you are using the python-object storage feature in YAML.

For a OS agnostic approach to carriage character assignment remember to use os.linesep.

Best of luck. Hope this helps.

You have a bit of redundancy here if you call close on f after the context manager exited. Not sure about the flush call, but I would think the context manager takes care of this as well. — Darkonaut, Jan 02 '18 at 23:31
Thanks for catching that. Honestly, flush() is just a force of habit at this point. And I always forget if/where there should be a close(). Thanks again. — Dan Temkin, Jan 02 '18 at 23:53

score 0 · Answer 4 · answered Oct 23 '18 at 12:24

Try out this method, i am using for updating yaml or json files. def update_dictionary_recursively(dictionary, key, value, key_separator="."): """Update givendictionarywith the givenkeyandvalue`.

if dictionary contains value as dict E.g. {key1:value1, key2:{key3, {key4:value4}}} and you have to
update key4 then `key` should be given as `key2.key3.key4`.

If dictionary contains value as list E.g. {key1:{key2:[{key3:valie3}, {key4:value4}]}} and you have to update
key4 then `key` should be given as `key1.key2[1].key4`.

:param dictionary: Dictionary that is to be updated.
:type dictionary: dict
:param key: Key with which the dictionary is to be updated.
:type key: str
:param value: The value which will be used to update the key in dictionary.
:type value: object
:param key_separator: Separator with which key is separated.
:type key_separator str
:return: Return updated dictionary.
:rtype: dict
"""
index = key.find(key_separator)
if index != -1:
    current_key = key[0:index]
    key = key[index + 1:]
    try:
        if '[' in current_key:
            key_index = current_key.split('[')
            current_key = key_index[0]
            list_index = int(key_index[1].strip(']'))
            dictionary[current_key][list_index] = update_dictionary_recursively(
                dictionary[current_key][list_index], key, value, key_separator)
        else:
            dictionary[current_key] = update_dictionary_recursively(dictionary[current_key],
                                                                                    key, value, key_separator)
    except (KeyError, IndexError):
        return dictionary
else:
    if '[' in key:
        key_index = key.split('[')
        list_index = int(key_index[1].strip(']'))
        if list_index > len(dictionary) - 1:
            return dictionary
        dictionary[list_index] = value
    else:
        if key not in dictionary:
            return dictionary
        dictionary[key] = value
return dictionary

`

Update YAML file programmatically

4 Answers4

testyaml.py

Resulting data.yaml

`testyaml.py`

Resulting `data.yaml`