Load YAML nested with Jinja2 in Python

Question

I have a YAML file (all.yaml) that looks like:

...
var1: val1
var2: val2
var3: {{var1}}-{{var2}}.txt
...

If I load it in Python like this:

import yaml

f = open('all.yaml')
dataMap = yaml.safe_load(f)
f.close()
print(dataMap["var3"])

the output is {{var1}}-{{var2}}.txt and not val1-val2.txt.

Is it possible to replace the nested vars with the value?

I tried to load it with:

import jinja2
templateLoader = jinja2.FileSystemLoader( searchpath="/path/to/dir" )
templateEnv = jinja2.Environment( loader=templateLoader )
TEMPLATE_FILE = "all.yaml"
template = templateEnv.get_template( TEMPLATE_FILE )

The exception is no longer thrown, now I am stuck and have to research how to proceed.

Answer 1

First define an Undefined class and load yaml to get known values. Then load it again and render with known values.

#!/usr/bin/env python

import yaml
from jinja2 import Template, Undefined

str1 = '''var1: val1
var2: val2
var3: {{var1}}-{{var2}}.txt
'''

class NullUndefined(Undefined):
  def __getattr__(self, key):
    return ''

t = Template(str1, undefined=NullUndefined)
c = yaml.safe_load(t.render())

print t.render(c)

Run it:

$ ./test.py
var1: val1
var2: val2
var3: val1-val2.txt

Answer 2

Here is one possible solution:

1. Parse your YAML document with the yaml module
2. Iterate over the keys in your YAML document, treating each value as a Jinja2 template to which you pass in the keys of the YAML document as parameters.

For example:

import yaml
from jinja2 import Template

with open('sample.yml') as fd:
    data = yaml.load(fd)

for k, v in data.items():
    t = Template(v)
    data[k] = t.render(**data)

print yaml.safe_dump(data, default_flow_style=False)

This will work fine with your particular example, but wouldn't do anything useful for, say, nested data structures (in fact, it would probably just blow up).

Answer 3

There is no replacement/substitution of scalar parts within the YAML specification.

Anything you want to do on that level has to be done in your application. For me, and for YAML, {{var1}} is just a nested mapping. {{var1}} is short for {{var1: null}: null}. After that the - is not allowed.

There are however multiple problems with your post:

1. You are using PyYAML which only supports the old (2005) YAML 1.1. Therefore you cannot you cannot have multiple documents (i.e. ended with ...) without using an explicit document start (---) like you can in YAML 1.2

2. Even if you correct the first line to read --- instead of ... your file will not load as a dict {{var1}} cannot be followed by a scalar - (from -{{var2}}.txt)

3. And if you would just use {{var1}} in your file, PyYAML cannot load this as it loads YAML mappings as Python dict and Python doesn't allow mutable keys for a dict. Just like you get an TypeError in Python when you try to do: {dict(var1=None): None}

So you should at least change your input file all.yaml to:

---
var1: val1
var2: val2
var3: '{{var1}}-{{var2}}.txt'
...

to get this to load in YAML.

You'll have to load this file two times:

• once by PyYAML to get the values that you can use to render template
• once as template by jinja2

After you render the template you load that (string) once more in PyYAML and you have the value that you want.

Given the corrected all.yaml as specified above in the current directory and this program:

import yaml
import jinja2

YAML_FILE = 'all.yaml'
with open(YAML_FILE) as fp:
    dataMap = yaml.safe_load(fp)

env = jinja2.Environment(loader=jinja2.FileSystemLoader(searchpath='.'))
template = env.get_template(YAML_FILE)

data = yaml.safe_load(template.render(**dataMap))
print(data["var3"])

will print what you wanted:

val1-val2.txt

Answer 4

I do not believe you can use:

yaml.load 

or

yaml.safe_load 

on a file containing jinja2 variables as values. The {{variable}} will attempt to be interpreted as a dict by yaml.