Update Yaml with jinja 2 variables in it

Question

I have a use case similar to this post: How can I update a .yml file, ignoring preexisting Jinja syntax, using Python?

I tried to follow the answers given in that but none of those working for my YAML syntax. Following is my YAML:

kind: Service
metadata:
  name: {{ .Chart.Name }}
  labels:
    app: {{ .Chart.Name }}
    service: {{ .Chart.Name }}
spec:
{{- if eq .Values.global.node.networkIpVersion "ipv6" }}
  ipFamily: IPv6
{{- end }}
  type: ClusterIP
  ports:
  - name: http2-8080
    protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: {{ .Chart.Name }}
---
apiVersion: v1
kind: Service
metadata:
  name: {{ .Chart.Name }}-traffic
  labels:
    app: {{ .Chart.Name }}
    service: {{ .Chart.Name }}-traffic
spec:
  ports:
  - name: http-2112
    port: {{ .Values.global.monitor.prometheusExporter.trafficListenPort }}
    protocol: TCP
    targetPort: 2112
  selector:
    app: {{ .Chart.Name }}

Another thing is my YAML has more than one template in it.

My use case is more for Node JS but as I couldn't find any solution in that language so I turned to Python.

If your use-case is to modify the shown input without interpreting the templating instructions, then you need a data model that can represent that structure including the templating instructions. Generally, your input is not valid YAML before you process it with Jinja so you cannot load it unprocessed with a YAML parser. OTOH, Jinja is an interpreter, and to my knowledge does not provide API access to a template's structure – and even if it did, the non-Jinja parts would be disconnected strings that might not parse as standalone YAML fragments. This is a case of using wrong tools for the job. — flyx, Sep 23 '21 at 12:14
Now the question is, what would be the right tools? There are possibilities, but no definite answer. XML, of all things, has the possibility of representing processing instructions in its DOM. You can also try to encode your templating instructions directly in YAML instead of using Jinja, which is generally possible with local tags (e.g. `!expr .Chart.Name` instead of `{{ .Chart.Name}}`) but quickly gets messy when you want to concatenate strings to a value. You might want to edit your question to detail your use-case more, but this is, in general, a problem too complex for SO. — flyx, Sep 23 '21 at 12:18

Anthon · Accepted Answer · 2021-09-23T13:16:04.097

The problem is that the answer you link to, as well as the version of ruamel.yaml.jinja2 that you are using do not know about the jinja2's if-statement. During loading the {{ part of {{- is replaced so it can be recognised as YAML, but that causes parsing issues (because it isn't indented properly nor a key-value pair). Instead it should be turned into some form of comment, as is done in ruamel.yaml.jinja2 version 0.2.6.

With that you can load, update and dump this multidocument YAML file:

import os
import sys
import pathlib
import ruamel.yaml


input = pathlib.Path('input.yaml')

yaml = ruamel.yaml.YAML(typ='jinja2')

print('ruamel.yaml.jinja2 version:', ruamel.yaml.jinja2.__version__)
print('====================')
sys.stdout.flush()
data = list(yaml.load_all(input))
# print(data)
data[0]['spec']['ports'][0]['port'] = 1234  # first document update
data[1]['spec']['ports'][0]['targetPort'] = 5678  # second document update
yaml.dump_all(data, pathlib.Path('output.yaml'))

os.system('diff -u input.yaml output.yaml')

which gives:

ruamel.yaml.jinja2 version: 0.2.6
====================
--- input.yaml  2021-09-23 14:55:42.303722955 +0200
+++ output.yaml 2021-09-23 15:13:14.663716699 +0200
@@ -12,7 +12,7 @@
   ports:
   - name: http2-8080
     protocol: TCP
-    port: 8080
+    port: 1234
     targetPort: 8080
   selector:
     app: {{ .Chart.Name }}
@@ -29,6 +29,6 @@
   - name: http-2112
     port: {{ .Values.global.monitor.prometheusExporter.trafficListenPort }}
     protocol: TCP
-    targetPort: 2112
+    targetPort: 5678
   selector:
     app: {{ .Chart.Name }}

RIght now its saying the latest version is 0.2.4 although I can see the updated one on pypi.org. Not sure if it takes time to reflect through pip — user3170450, Sep 23 '21 at 15:55
I am able to install it, some config issue on my end and it worked as well. Now I have another question: We are parsing here helm bundles which may contain any file with any kind of jinja variables. So can I expect this library will handle all those scenarios or there are still certain kinds which are not yet supported ? — user3170450, Sep 23 '21 at 16:13
I originally only set it up for `{{ }}` style variables. I never looked at jinja2 and don't know what its full syntax is. — Anthon, Sep 23 '21 at 16:31

Update Yaml with jinja 2 variables in it

1 Answers1