Print keys in yaml, so that they can be used for jsonpath

Question

echo "apiVersion: v1
kind: Node
metadata:
  name: host-cluster-control-plane-64j47
  labels:
    beta.kubernetes.io/arch: amd64
" | yq -o p

Result:

apiVersion = v1
kind = Node
metadata.name = host-cluster-control-plane-64j47
metadata.labels.beta.kubernetes.io/arch = amd64

That's almost what I want. I am looking for the key to get values.

I could use metadata.name like this:

echo "apiVersion: v1
kind: Node
metadata:
  name: host-cluster-control-plane-64j47
  labels:
    beta.kubernetes.io/arch: amd64
" | yq '.metadata.name'

But the -o p option of yq does not quote the key, if needed.

I can't use metadata.labels.beta.kubernetes.io/arch as key, since the correct syntax is metadata.labels["beta.kubernetes.io/arch"].

Is there an automated way to get the keys of a yaml file so that I can use the keys in yq (or jq)?

The desired output would be something like this:

apiVersion = v1
kind = Node
metadata.name = host-cluster-control-plane-64j47
metadata.labels["beta.kubernetes.io/arch"] = amd64

I am looking for the valid key, because I want to create a second command line to select these values.

For example:

❯ k get nodes -o yaml | yq '.items[].metadata.labels["beta.kubernetes.io/arch"]'

amd64
amd64
amd64

Can you elaborate on "the correct syntax"? What (exact) criteria disambiguates the ways how level `metadata` and level `labels` are glued together (here `A.B`) from how level `labels` and level `beta.kubernetes.io/arch` are (here `A["B"]`). Is it only the presence of dots (and/or maybe the slash) in "B"? What if these are present in "A" (on top-level)? Please define or include more samples. — pmf, Mar 27 '23 at 19:07
@pmf I added this to the question: I am looking for the valid key, because I want to create a second commadn line to select these values. For example ... Is it more clear now? — guettli, Mar 28 '23 at 08:53
Yes, apparently you want to generate code parseable by yq (although I still can't follow why you want the dot omitted on top-level). However, this looks like an [XY problem](https://xyproblem.info/). If your ultimate goal is to retrieve (or otherwise process) values at given paths, rather focus on extracting pure data (the paths in this case) in a post-processable way (e.g. `yq '.. | select(tag == "!!str") | path | . style="flow"'` or `yq '.. | select(tag == "!!str") | [{"path": path, "value": .}]'`), and then operate on/with that instead. Embedding generated code is rarely a good idea. — pmf, Mar 28 '23 at 10:03
Agree with @pmf if what you are going to process the path with `yq` output the path array and then you can later use the setpath path operation... — mike.f, Mar 28 '23 at 23:26

score 5 · Accepted Answer · answered Mar 28 '23 at 03:55

You can get close by doing something like:

yq '(.. | key | select(test("\."))) |= ("[\"" + . + "\"]")' file.yaml -op

apiVersion = v1
kind = Node
metadata.name = host-cluster-control-plane-64j47
metadata.labels.["beta.kubernetes.io/arch"] = amd64

Or you could do:

yq '(.. | key | select(test("\."))) |= sub("\.", "\.")' file.yaml -op

apiVersion = v1
kind = Node
metadata.name = host-cluster-control-plane-64j47
metadata.labels.beta\\.kubernetes\\.io/arch = amd64

BTW - I'm not sure how it's supposed be escaped in property files, I'd be willing to update yq to do it natively someone raises a bug with details on github...

Disclaimer: I wrote yq

score 3 · Answer 2 · answered Mar 25 '23 at 03:42

Be forewarned that you're trying to abuse the half-baked yq -o=props output but it will not do what you want even if it was correctly implemented¹ because JSONPath uses \. to escape those dots, even though that may be a kubectl-ism since that syntax isn't documented:

$ kubectl get no -o jsonpath='{.items[*].metadata.labels.beta\.kubernetes\.io/arch}'
amd64

It may be possible to do special handling for annotations:, labels:, selector:, and other free-form key-value pairs in the kubernetes objects that interest you, but for sure it's not going to work in the general case the way you wanted it to

apologies for the jq in this, I don't know how to do this in yq and of course you can use any placeholder character you'd like, I just know tab characters cannot appear in kubernetes annotations nor labels

$ echo '
apiVersion: v1
kind: Node
metadata:
  labels:
    beta.kubernetes.io/arch: amd64
' | yq -o=json | jq '
.metadata.labels = (
  [.metadata.labels|to_entries[]
  | {key: .key|gsub("[.]";"\t"), value}
  ]
  |from_entries)' | yq -o=p | sed -Ee 's@\\t@\\.@g'
apiVersion = v1
kind = Node
metadata.labels.beta\.kubernetes\.io/arch = amd64

fn 1: someone else raised that same concern and it went just as poorly, and I said "incorrectly implemented" because echo '{"alpha.beta":{"charlie":1}}' | yq -o=p | yq -p=p does not round trip

It seems the properties library does not allow escaping .. I even tried being sneaky with beta\u002Ekubernetes but it seems the unicode escape processing happens before the key lexing does and thus suffers from the same bug

Thank you for your detailed answer. It seems there is no perfect solution to my question. `yq -o p` works for my case. I can do the necessary quoting by hand. I was just curious if there is a tool which can help me to address the needed keys. — guettli, Mar 25 '23 at 06:11

djmonki · Answer 3 · 2023-04-08T21:20:11.740

PYTHON

Aware that the ask is yq, however as an out of the box thinking option, the keys can be printed using a python script. Based on Python 3, using Python modules argparse and pyyaml, installed using pip3

To execute script: python3 yaml_to_json.py <file>

Script: yaml_to_json.py

import argparse
import yaml

def print_keys(data, prefix=""):
    if isinstance(data, dict):
        for key, value in data.items():
            new_prefix = f"{prefix}.{key}" if prefix else key
            if isinstance(value, dict):
                print_keys(value, new_prefix)
            elif isinstance(value, list):
                for i, item in enumerate(value):
                    print_keys(item, f"{new_prefix}[{i}]")
            else:
                if "." in key:
                    print(f'{new_prefix}["{key}"] = {value}')
                else:
                    print(f'{new_prefix} = {value}')

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Print keys in a YAML file for JSONPath")
    parser.add_argument("yaml_file", type=str, help="path to YAML file")
    args = parser.parse_args()

    with open(args.yaml_file, "r") as f:
        data = yaml.safe_load(f)
        for key, value in data.items():
            if isinstance(value, dict):
                for k, v in value.items():
                    if isinstance(v, dict):
                        for sub_k, sub_v in v.items():
                            new_key = f"{k}[\"{sub_k}\"]"
                            print(f"metadata.{new_key} = {sub_v}")
                    else:
                        new_key = f"metadata.{k}"
                        print(f"{new_key} = {v}")
            else:
                print(f"{key} = {value}")

The result is:

apiVersion = v1
kind = Node
metadata.name = host-cluster-control-plane-64j47
metadata.labels["beta.kubernetes.io/arch"] = amd64

Explanation:

The 1st block defines a function to print the keys that will recursively traverse the data and print the keys/values in the format for use with JSONPath.
The 2nd block defines argparse parser to parse the command line argument to the file.
The 3rd block opens the file, loads the data using yaml.safe_load(), traverses the data to print the keys. The metadata prefix is added to the keys to match the desired output format.

Print keys in yaml, so that they can be used for jsonpath

3 Answers3

PYTHON