Replace values from YAML in JSON file - Python 3

Question

I want to replace the values from YAML file into a JSON file using Python 3.

I have many JSON files where I want to get the values from a master YAML file and replace only certain values like source server ip, email, hostname.

E.g. I have this YAML file (mast_conf.yaml):

-  sourcesystem1:
    sourceServer: 1.2.3.500
    MailTo: gokul@gmail.com
-  sourcesystem2:
    sourceServer1: 2.2.3.500
    sourceServer2: 3.2.3.500
    MailTo: gokul@gmail.com

A JSON file (sourcesystem1.json):

{
    "source":"sourcesystem1",
    "frequency":"daily",
    "sourceServer":"1.2.1.2",
    "hostName":"1.2.1.3",
    "fileFormat":"csv",
    "delimiterType":"semicolon"
}

Another JSON file (sourcesystem2.json):

{
    "source":"sourcesystem2",
    "frequency":"daily",
    "sourceServer":"1.2.3.2",
    "hostName":"1.2.1.7",
    "fileFormat":"csv",
    "delimiterType":"commaseperated"
}

Below is my code I am trying out to parse the value from the json file

import json
import yaml


with open("master_conf.yaml", 'r') as f:
yaml_config = yaml.safe_load(f)


yaml_config = {
list(config.keys()[0]): list(config[config.keys()[0]])
for config in yaml_config
}


json_files = ( "sourcesystem1.json",
"sourcesystem2.json",
)


for json_file in json_files:
with open(json_file, "r") as f:
sourcesystem_conf = json.load(f)


sourcesystem = sourcesystem_conf["source"]


if sourcesystem in yaml_config:
for key, value in yaml_config[sourcesystem].items():
sourcesystem_conf[key] = value


with open(json_file, "w") as f:
json.dump(sourcesystem_conf, f, indent=2)

I am getting the below error by program

TypeError: 'dict_keys' object does not support indexing

When I run indivudually I get this issue for yaml

>>> yaml_config = { ... config.keys()[0]: config[config.keys()[0]] ... for config in yaml_config ... } Traceback (most recent call last): File "<stdin>", line 3, in <module> File "<stdin>", line 3, in <dictcomp> TypeError: 'dict_keys' object is not subscriptable >>>

Is there easier method to achieve my end goal where I want to replace the values in the JSON file from the Yaml configuration file

This is needed to update 1000s of Json file in a automated way for updating it from a master Yaml file

Answer 1

The easiest way is to use pyyaml, see Jon's answer.

Then you can load you yaml file using it :

>>> import yaml
>>> yaml_config = yaml.safe_load(yaml_file)
>>> yaml_config
[{'sourcesystem1': {'MailTo': 'gokul@gmail.com', 'sourceServer': '1.2.3.500'}},
 {'sourcesystem2': {'MailTo': 'gokul@gmail.com',
   'sourceServer1': '2.2.3.500',
   'sourceServer2': '3.2.3.500'}}]

It will be easier to manipulate a dict with source systems as keys.

In python 2 aDict.keys() returns a list so the following will work :

>>> yaml_config = {
    config.keys()[0]: config[config.keys()[0]]
    for config in yaml_config
}
>>> yaml_config
{'sourcesystem1': {'MailTo': 'gokul@gmail.com', 'sourceServer': '1.2.3.500'},
 'sourcesystem2': {'MailTo': 'gokul@gmail.com',
  'sourceServer1': '2.2.3.500',
  'sourceServer2': '3.2.3.500'}}

In python 3 aDict.keys() no longer returns a list so you can simply use a for loop :

yaml_config = {}
for config in yaml_config_raw:
    source = [key for key in config][0]
    yaml_config[source] = config[source]

Then you can just iterate over your json files to update them :

import json
import yaml

with open("mast_conf.yaml", 'r') as f:
    yaml_config_raw = yaml.safe_load(f)


yaml_config = {}
for config in yaml_config_raw:
    source = [key for key in config][0]
    yaml_config[source] = config[source]

json_files = (
    "sourcesystem1.json",
    "sourcesystem2.json",
)

for json_file in json_files:
    with open(json_file, "r") as f:
        sourcesystem_conf = json.load(f)

    sourcesystem = sourcesystem_conf["source"]

    if sourcesystem in yaml_config:
        for key, value in yaml_config[sourcesystem].items():
            sourcesystem_conf[key] = value

    with open(json_file, "w") as f:
        json.dump(sourcesystem_conf, f, indent=2)