Changing dict content dynamically using Python

Question

Say, I want to dynamically edit a Kubernetes deployment file that looks like this using Python:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 4
  selector:
    matchLabels:
      app: guestbook
      tier: frontend
  template:
    metadata:
      labels:
        app: guestbook
        tier: frontend
    spec:
      containers:
      - env:
        - name: GET_HOSTS_FROM
          value: dns
        image: gcr.io/google-samples/gb-frontend:v4
        name: php-redis
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 100Mi

I have a code that opens this yaml file where I want to change the content of spec.replicas branch from 2 to 4:

 with open(deployment_yaml_full_path, "r") as stream:
        try:
            deployment = yaml.safe_load(stream)
            if value_to_change[0] == 'spec.replicas':
                deployment['spec']['replicas'] = value_to_change[1]
        except yaml.YAMLError as exc:
            logger.error('There was a problem opening a deployment file in path: ',
                         deployment_yaml_full_path=deployment_yaml_full_path, exc=exc)

I would like to know if there's a way to avoid the hardcoded part here to something more dynamic:

if value_to_change[0] == 'spec.replicas':
                deployment['spec']['replicas'] = value_to_change[1]

Is there a way?

Answer 1

target = value_to_change[0].split('.')

if len(target) == 2 and target[0] in deployment and target[1] in deployment[target[0]]:
    deployment[target[0]][target[1]] = value_to_change[1]

If the paths can be longer:

path_len = len(target)

d = deployment
path_exists = True
for i in range(path_len):
  if target[i] in d:
    d = d[target[i]]
  else:
    path_exists = False

if path_exists:
   d = value_to_change[1]

Answer 2

I believe you want to change the YAML to JSON/dictionary using PyYaml

import yaml
import json

with open('config.yml', 'r') as file:
    configuration = yaml.safe_load(file)

with open('config.json', 'w') as json_file:
    json.dump(configuration, json_file)
    
output = json.dumps(json.load(open('config.json')), indent=2)
print(output)

After that you would like to use:

class obj(object):
def __init__(self, d):
    for k, v in d.items():
        if isinstance(k, (list, tuple)):
            setattr(self, k, [obj(x) if isinstance(x, dict) else x for x in v])
        else:
            setattr(self, k, obj(v) if isinstance(v, dict) else v)

Usage Example:

>>> d = {'a': 1, 'b': {'c': 2}, 'd': ["hi", {'foo': "bar"}]}
>>> x = obj(d)
>>> x.b.c
2
>>> x.d[1].foo
'bar'

The last phase will be to change the value by string path:

from collections.abc import MutableMapping

def set_value_at_path(obj, path, value):
    *parts, last = path.split('.')

    for part in parts:
        if isinstance(obj, MutableMapping):
            obj = obj[part]
        else:
            obj = obj[int(part)]

    if isinstance(obj, MutableMapping):
        obj[last] = value
    else:
        obj[int(last)] = value

Answer 3

Disclaimer: this is solely for solace, fun and educational purpose

I think the correct way to do what you want is to study json path, there is an easy example here and this stupid answer of mine could help you create the actual json path expressions!

Well, this is the one-liner you should NOT use to achieve what you want in the most dynamic way possible:

exec(f'deployment{"".join([f"[##{val}##]" for val in value_to_change[0].split(".")])}={value_to_change[1]}'.replace('##','"'))

We create a list from the value_to_change[0] value splitting by dot.

  value_to_change[0].split(".")

We get each val in this list end we enclose it in "dictionary" syntax (hashtags.. well they can be anything you want, but f-strings do not support backslashes, hence I replace hashtags with the quotes after)

[f"[##{val}##]" for val in value_to_change[0].split(".")]

We join the vals in a string and replace the hashtags and add the deployment string

f'deployment{"".join([f"[##{val}##]" for val in value_to_change[0].split(".")])}={value_to_change[1]}'.replace('##','"')

The result will be this string... (what you would normally write to change that value):

'deployment["spec"]["replicas"]=50'

We perform actual monstrosity executing the string.
In my example value_to_change[1] is 50

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "frontend"
  },
  "spec": {
    "replicas": 50,
    "selector": {
      "matchLabels": {
        "app": "guestbook",
        "tier": "frontend"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "app": "guestbook",
          "tier": "frontend"
        }
      },
      "spec": {
        "containers": [
          {
            "env": [
              {
                "name": "GET_HOSTS_FROM",
                "value": "dns"
              }
            ],
            "image": "gcr.io/google-samples/gb-frontend:v4",
            "name": "php-redis",
            "ports": [
              {
                "containerPort": 80
              }
            ],
            "resources": {
              "requests": {
                "cpu": "100m",
                "memory": "100Mi"
              }
            }
          }
        ]
      }
    }
  }
}

Have FUN!

Answer 4

The json path solution is like this

You just need parse function from jsonpath_ng library (pip install jsonpath_ng)

from jsonpath_ng import parse

# create the expression
expr = parse(f'$.{".".join(value_to_change[0].split("."))}')
# you can check if it works with expr.find(deployment) more after.

# actually change the value
expr.update(deployment, value_to_change[1])

# deployment['spec']['replicas'] now has changed

When you check with find you will see a lot of output, just focus on the first part:

expr.find(deployment)
>>> [DatumInContext(value=4, path=Fields('replicas')

As you can see the expression is really easy: "$.path.to.value.you.want"
If you wonder how to manage list inside, the syntax is the same as python [], with inside the index, or * to say all the items in the list (and this is priceless!!)

This is a very good place to learn everything you need!