Change in a yaml file python

Question

I have to replace tags like these in a yaml file.
The data that used to be replaced is stored in a list.
I have this YAML file

vagrantfile:    
    target: local
    vm:
        provider:
            local:
                box: puphpet/ubuntu1404-x<BITS> # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_url: puphpet/ubuntu1404-x<BITS> # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_version: '0'
                chosen_virtualizer: <virtualiser>  # MEERKEUZE: virtualbox OF vmware, STANDAARD 64
                virtualizers:
                    virtualbox:
                        modifyvm:
                            natdnshostresolver1: false
                        showgui: 0
                    vmware:
                        numvcpus: 1
                    parallels:
                        use_linked_clone: 0
                        check_guest_tools: 0
                        update_guest_tools: 0
                machines:
                    vflm_7orc03npv15g:
                        id: <ID> # STRING
                        hostname: <HOSTNAME> # STRING
                        network:
                            private_network: <IP-ADDRESS> #lOKAAL IP, STANDARAARD 192.168.50.101
                            forwarded_port:
                                hon1d:
                                    host: '80'
                                    guest: '80'
                        memory: '<MEGABYTES>' # AANTAL, STANDAARD VEELVOUD VAN 1024
                        cpus: '<CORES>' # AANTAL CPUS, STANDAARD AANTAL CORES - 1, NIET HOGER DAN AANTAL CORES

What can I do to change al these tags?

**See also:** https://stackoverflow.com/questions/41620674/use-placeholders-in-yaml — dreftymac, Nov 08 '17 at 23:42

Anthon · Answer 1 · 2016-05-18T16:13:22.667

You should do this at the level of YAML, just to make sure that if your substitution forces the resulting string to be quoted that this actually happens. E.g. if your <ID> needs to be replaces by something like @abc the actual line in the YAML file needs to become

id: '@abc'

and not

id: '@abc'

as @ is reserved. There are several such caveats, so substituting on the string level is not a good idea.

You can do:

import sys
import ruamel.yaml

yaml_str = """\
vagrantfile:
    target: local
    vm:
        provider:
            local:
                box: puphpet/ubuntu1404-x<BITS> # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_url: puphpet/ubuntu1404-x<BITS> # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_version: '0'
                chosen_virtualizer: <virtualiser>  # MEERKEUZE: virtualbox OF vmware, STANDAARD 64
                virtualizers:
                    virtualbox:
                        modifyvm:
                            natdnshostresolver1: false
                        showgui: 0
                    vmware:
                        numvcpus: 1
                    parallels:
                        use_linked_clone: 0
                        check_guest_tools: 0
                        update_guest_tools: 0
                machines:
                    vflm_7orc03npv15g:
                        id: <ID> # STRING
                        hostname: <HOSTNAME> # STRING
                        network:
                            private_network: <IP-ADDRESS> #lOKAAL IP, STANDARAARD 192.168.50.101
                            forwarded_port:
                                hon1d:
                                    host: '80'
                                    guest: '80'
                        memory: '<MEGABYTES>' # AANTAL, STANDAARD VEELVOUD VAN 1024
                        cpus: '<CORES>' # AANTAL CPUS, STANDAARD AANTAL CORES - 1, NIET HOGER DAN AANTAL CORES
"""


def replace(data, values):
    def do_one(data, values):
        if isinstance(data, dict):
            for k in data:
                data[k] = do_one(data[k], values)
            return data
        elif isinstance(data, list):
            for idx, elem in enumerate(data):
                data[idx] = do_one(elem, values)
            return data
        elif isinstance(data, str):
            for k in values:  # full match
                if data == k:
                    return values[k]
            if '<' in data and '>' in data:
                for k in values:
                    data = data.replace(k, str(values[k]))
                return data


    expanded = { ('<' + k + '>'): v for (k, v) in values.iteritems()}
    do_one(data, expanded)

data = ruamel.yaml.round_trip_load(yaml_str)

replace(data, {
    'BITS': 64,
    'virtualiser': 'virtualbox',
    'HOSTNAME': 'localhost',
    'IP-ADDRESS': '192.168.0.1',
    'ID': '@abc',
    'MEGABYTES': 2048,
    'CORES': 8,
    })

ruamel.yaml.round_trip_dump(data, stream=sys.stdout, indent=4)

which will give you:

vagrantfile:
    target:
    vm:
        provider:
            local:
                box: puphpet/ubuntu1404-x64     # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_url: puphpet/ubuntu1404-x64     # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_version:
                chosen_virtualizer: virtualbox     # MEERKEUZE: virtualbox OF vmware, STANDAARD 64
                virtualizers:
                    virtualbox:
                        modifyvm:
                            natdnshostresolver1:
                        showgui:
                    vmware:
                        numvcpus:
                    parallels:
                        use_linked_clone:
                        check_guest_tools:
                        update_guest_tools:
                machines:
                    vflm_7orc03npv15g:
                        id: '@abc' # STRING
                        hostname: localhost  # STRING
                        network:
                            private_network: 192.168.0.1  #lOKAAL IP, STANDARAARD 192.168.50.101
                            forwarded_port:
                                hon1d:
                                    host:
                                    guest:
                        memory: 2048          # AANTAL, STANDAARD VEELVOUD VAN 1024
                        cpus: 8         # AANTAL CPUS, STANDAARD AANTAL CORES - 1, NIET HOGER DAN AANTAL CORES

Please note that:

the comments are preserved
that the value for cpus and memory are integers, not strings
that the value for id gets properly quoted as necessary.

The above uses ruamel.yaml: disclaimer I am the author of that package. You can do the same with PyYAML, but you will lose the comments and you need to restrict yourself to the YAML 1.1 standard (from 2005) and not the YAML 1.2 standard (from 2009)

score 0 · Answer 2 · answered May 18 '16 at 13:19

0

Probably easiest to use pyyaml. E.g. to set the cpus value:

import yaml
foo = yaml.load(open('file.yml', 'r'))
foo['vagrantfile']['vm']['provider']['local']['machines']['vflm_7orc03npv15g']['memory'] = 'bar'
yaml.dump(foo, open('out.yml', 'w'))

Only issue is you'll lost the comments this way.

answered May 18 '16 at 13:19

radpotato

1,332
1
12
20

Shouldn't there be an option to iterate the file and replace the tags? – Robin Deblaere May 18 '16 at 13:41
because i'm using python 2.7 – Robin Deblaere May 18 '16 at 13:44
Ah, I guess I didn't realise the tags were always going to be the same. In that case I'd go with something like gipsy's solution with a string template (i'd personally use curly braces and `.format`). – radpotato May 18 '16 at 14:30

score 0 · Answer 3 · answered May 18 '16 at 13:27

You can convert the yaml into a string template and use string formatting to achieve this

template = """vagrantfile:    
    target: local
    vm:
        provider:
            local:
                box: puphpet/ubuntu1404-x%(BITS)s # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_url: puphpet/ubuntu1404-x%(BITS)s # MEERKEUZE: 32 OF 64, STANDAARD 64
                box_version: '0'
                chosen_virtualizer: %(virtualiser)s  # MEERKEUZE: virtualbox OF vmware, STANDAARD 64
                virtualizers:
                    virtualbox:
                        modifyvm:
                            natdnshostresolver1: false
                        showgui: 0
                    vmware:
                        numvcpus: 1
                    parallels:
                        use_linked_clone: 0
                        check_guest_tools: 0
                        update_guest_tools: 0
                machines:
                    vflm_7orc03npv15g:
                        id: %(ID)s # STRING
                        hostname: %(OSTNAME)s # STRING
                        network:
                            private_network: %(IP-ADDRESS)s #lOKAAL IP, STANDARAARD 192.168.50.101
                            forwarded_port:
                                hon1d:
                                    host: '80'
                                    guest: '80'
                        memory: '%(MEGABYTES)s' # AANTAL, STANDAARD VEELVOUD VAN 1024
                        cpus: '%(CORES)s' # AANTAL CPUS, STANDAARD AANTAL CORES - 1, NIET HOGER DAN AANTAL CORES"""


replaced = template % {"BITS": "some value", "virtualiser": "something", "ID": 2412, "OSTNAME": "OS name", "IP-ADDRESS": "some ip", "MEGABYTES": 100, "CORES": 16}

Depending on the values that you substitute this can result in output that cannot be parsed by a YAML parser. — Anthon, May 18 '16 at 15:55

Change in a yaml file python

3 Answers3

Linked