parse and update YAML file at particular position in python

Question

My yaml file looks like below:

jobs:
- build_logs_to_retain: 1000
  name: demo-comp2
  plan:
  - get: landscape
    passed:
    - demo-comp1
    trigger: true
  - attempts: 3
    config:
      container_limits: {}
      image_resource:
        source:
          password: ""
          repository: ubuntu
          username: ""
        type: docker-image
      inputs:
      - name: landscape
      params:
        INFLUXDB_DATABASE: newdatabase
        INFLUXDB_PASSWORD: 123456789abcdefgh
        INFLUXDB_URL: influxdb.com
        INFLUXDB_USER: newuser
        GITHUB_PRIVATE_KEY: |
          -----BEGIN RSA PRIVATE KEY-----
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          M8rG0u9pa6rdALV1DzV2jM5EwFg8w8KkBz+iXBZty4oKYYGirJ26h/XtJmJUCAwE
          HEn0KXVI/8myJLcBIRE7QCOKJI76SzBtkNShZimN3nGfoecoVR5zlJdHDNbJ7LsL
          -----END RSA PRIVATE KEY-----
      platform: linux
      run:
        path: my_path/my_path/script.sh
    task: task1
    timeout: 120m
  - attempts: 3
    config:
      container_limits: {}
      image_resource:
        source:
          password: ""
          repository: ubuntu
          username: ""
        type: docker-image
      inputs:
      - name: landscape
      params:
        INFLUXDB_DATABASE: newdatabase
        INFLUXDB_PASSWORD: 123456789abcdefgh
        INFLUXDB_URL: influxdb.com
        INFLUXDB_USER: newuser
      platform: linux
      run:
        path: my_path/my_path/script.sh
    task: test-demo-comp2
    timeout: 20m
  serial: true
- build_logs_to_retain: 1000
  name: demo-comp1
  plan:
  - get: landscape
    trigger: true
  - attempts: 3
    config:
      container_limits: {}
      image_resource:
        source:
          password: ""
          repository: ubuntu
          username: ""
        type: docker-image
      inputs:
      - name: landscape
      params:
        INFLUXDB_DATABASE: newdatabase
        INFLUXDB_PASSWORD: 123456789abcdefgh
        INFLUXDB_URL: influxdb.com
        INFLUXDB_USER: newuser
      platform: linux
      run:
        path: my_path/my_path/script.sh
    task: deploy-demo-comp1
    timeout: 120m
  - attempts: 3
    config:
      container_limits: {}
      image_resource:
        source:
          password: ""
          repository: ubuntu
          username: ""
        type: docker-image
      inputs:
      - name: landscape
      params:
        INFLUXDB_DATABASE: newdatabase
        INFLUXDB_PASSWORD: 123456789abcdefgh
        INFLUXDB_URL: influxdb.com
        INFLUXDB_USER: newuser
      platform: linux
      run:
        path: my_path/my_path/script.sh
    task: test-demo-comp1
    timeout: 20m
  serial: true
resources:
- check_every: 5m
  name: landscape
  source:
    branch: master
    password: newpassword
    paths:
    - config/*
    - config/**/*
    uri: my_repo.git
    username: myuser
  type: git

I am trying to parse this YAML and add new key:values under params section. Currently params has below content:

INFLUXDB_DATABASE: newdatabase
INFLUXDB_PASSWORD: 123456789abcdefgh
INFLUXDB_URL: influxdb.com
INFLUXDB_USER: newuser

ANd i am trying to add new kay:value pair MY_NAME: Rohith; so that the content under params looks like below:

INFLUXDB_DATABASE: newdatabase
INFLUXDB_PASSWORD: 123456789abcdefgh
INFLUXDB_URL: influxdb.com
INFLUXDB_USER: newuser
MY_NAME: Rohith

I am trying this in python3; to parse itself, I am ending in too many for loops and if conditions. Could someone please help me here..! I have updated the yaml file by adding GITHUB_PRIVATE_KEY; the method by @RoadRunner fails to parse this GITHUB_PRIVATE_KEY. In the generated final YAML it tries to include single quotes and could see new lines in the GITHUB_PRIVATE_KEY

So basically you need to iterate over yaml content and extract keys named `params`. Then you may easily modify value for every such key, treating this value as a dict and adding new key-value pair `MY_NAME: Rohith` under it. As for ways for iteration, they are described in that question: https://stackoverflow.com/questions/9807634/find-all-occurrences-of-a-key-in-nested-dictionaries-and-lists. — Tsyvarev, Apr 03 '20 at 12:41

RoadRunner · Accepted Answer · 2020-04-03T15:59:34.337

You could use PyYAML to read the YAML file into a python object with safe_load():

import yaml

with open("data.yml") as yml_file:
    data = yaml.safe_load(yml_file)

You can install this library with pip install PyYAML.

Then you could create a recursive function to recurse the python object and update a dictionary with data you pass to it, given a search key:

def insert_data_rec(iterable, search_key, data):
    if isinstance(iterable, list):
        for item in iterable:
            if isinstance(item, (list, dict)):
                insert_data_rec(item, search_key, data)

    elif isinstance(iterable, dict):
        for k, v in iterable.items():
            if k == search_key:
                iterable[k].update(data)
            if isinstance(v, (list, dict)):
                insert_data_rec(v, search_key, data)

Then you could call this function to modify data and output the new contents with safe_dump():

insert_data_rec(data, search_key='params', data={'MY_NAME': 'Rohith'})

print(yaml.safe_dump(data))

Which Outputs the following data.

the final output has replaced the double-quotes with single quotes. here https://pastebin.com/yRafQiyK - i could see the original data.yml has double-quotes for few values which are replaced to single-quotes in the final output. — Rohith, May 10 '20 at 14:58
as this is due to `data = yaml.safe_load(yml_file)` - is there a way to tackle this. — Rohith, May 10 '20 at 15:38

parse and update YAML file at particular position in python

1 Answers1