jq - how to get all the values from a object within object

Question

I have the following json object.

{
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {
        "annotations": {
            "deployment.kubernetes.io/revision": "1",
        },
        "creationTimestamp": "2023-02-22T17:35:22Z",
        "generation": 1,
        "labels": {
            "app": "addresses",
            "group": "de.test",
            "provider": "jkube",
            "version": "2.3.0-3191"
        },
        "name": "addresses",
        "namespace": "linkerd",
        "resourceVersion": "910972",
        "uid": "aaee0eb0-5d66-44dd-b9b0-35e17bec4cf4"
    },
    "spec": {
        "progressDeadlineSeconds": 600,
        "replicas": 1,
        "revisionHistoryLimit": 2,
        "selector": {
            "matchLabels": {
                "app": "addresses",
                "group": "de.test",
                "provider": "jkube"
            }
        },
        "strategy": {
            "rollingUpdate": {
                "maxSurge": 1,
                "maxUnavailable": "50%"
            },
            "type": "RollingUpdate"
        },
        "template": {
            "metadata": {
                "annotations": {
                    "jkube.io/git-branch": "develop",
                    "jkube.io/git-commit": "77aaca38564dffff0ce10a8c70d4139b33f677b2",
                    "jkube.io/git-url": "http://bitbucket.org/testzentrale/test-monorepo",
                    "jkube.io/scm-tag": "HEAD",
                    "jkube.io/scm-url": "https://bitbucket.org/testzentrale/test-monorepo/test/addresses"
                },
                "creationTimestamp": null,
                "labels": {
                    "app": "addresses",
                    "group": "de.test",
                    "provider": "jkube",
                    "version": "2.3.0-3191"
                },
                "namespace": "test-testing"
            },
            "spec": {
                "containers": [
                    {
                        "env": [
                            {
                                "name": "SENTRY_ENVIRONMENT",
                                "valueFrom": {
                                    "fieldRef": {
                                        "apiVersion": "v1",
                                        "fieldPath": "metadata.namespace"
                                    }
                                }
                            },
                            {
                                "name": "SENTRY_SERVERNAME",
                                "valueFrom": {
                                    "fieldRef": {
                                        "apiVersion": "v1",
                                        "fieldPath": "metadata.name"
                                    }
                                }
                            },
                            {
                                "name": "SENTRY_TAGS",
                                "value": "service:addresses"
                            },
                            {
                                "name": "SENTRY_RELEASE",
                                "value": "2.3.0-3191"
                            },
                            {
                                "name": "JAVA_OPTIONS",
                                "value": "-Xms64m -Xmx384m"
                            },
                            {
                                "name": "KUBERNETES_NAMESPACE",
                                "valueFrom": {
                                    "fieldRef": {
                                        "apiVersion": "v1",
                                        "fieldPath": "metadata.namespace"
                                    }
                                }
                            },
                            {
                                "name": "HOSTNAME",
                                "valueFrom": {
                                    "fieldRef": {
                                        "apiVersion": "v1",
                                        "fieldPath": "metadata.name"
                                    }
                                }
                            }
                        ],
                        "envFrom": [
                            {
                                "configMapRef": {
                                    "name": "db-mssql-asd-connection"
                                }
                            },
                            {
                                "secretRef": {
                                    "name": "db-mssql-asd-connection"
                                }
                            },
                            {
                                "configMapRef": {
                                    "name": "db-postgre-asd-connection"
                                }
                            },
                            {
                                "secretRef": {
                                    "name": "db-postgre-asd-connection"
                                }
                            },
                            {
                                "configMapRef": {
                                    "name": "app-url"
                                }
                            },
                            {
                                "secretRef": {
                                    "name": "app-token"
                                }
                            }                                                      
                        ],
                        "image": "myrepo/test/addresses:testing-2.3.0-3191",
                        "imagePullPolicy": "IfNotPresent",
                        "livenessProbe": {
                            "failureThreshold": 3,
                            "httpGet": {
                                "path": "/",
                                "port": 8080,
                                "scheme": "HTTP"
                            },
                            "initialDelaySeconds": 300,
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "name": "spring-boot",
                        "ports": [
                            {
                                "containerPort": 8080,
                                "name": "http",
                                "protocol": "TCP"
                            },
                            {
                                "containerPort": 9779,
                                "name": "prometheus",
                                "protocol": "TCP"
                            },
                            {
                                "containerPort": 8778,
                                "name": "jolokia",
                                "protocol": "TCP"
                            }
                        ],
                        "readinessProbe": {
                            "failureThreshold": 2,
                            "httpGet": {
                                "path": "/actuator/health",
                                "port": 8080,
                                "scheme": "HTTP"
                            },
                            "initialDelaySeconds": 20,
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "resources": {
                            "limits": {
                                "memory": "1000Mi"
                            },
                            "requests": {
                                "cpu": "100m",
                                "memory": "500Mi"
                            }
                        },
                        "securityContext": {
                            "privileged": false
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "terminationMessagePolicy": "File",
                        "volumeMounts": [
                            {
                                "mountPath": "/deployments/config",
                                "name": "config"
                            }
                        ]
                    }
                ],
                "dnsPolicy": "ClusterFirst",
                "restartPolicy": "Always",
                "schedulerName": "default-scheduler",
                "securityContext": {},
                "terminationGracePeriodSeconds": 30,
                "volumes": [
                    {
                        "configMap": {
                            "defaultMode": 420,
                            "items": [
                                {
                                    "key": "application.properties",
                                    "path": "application.properties"
                                }
                            ],
                            "name": "addresses"
                        },
                        "name": "config"
                    }
                ]
            }
        }
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2023-02-22T17:35:23Z",
                "lastUpdateTime": "2023-02-22T17:35:23Z",
                "message": "Deployment does not have minimum availability.",
                "reason": "MinimumReplicasUnavailable",
                "status": "False",
                "type": "Available"
            },
            {
                "lastTransitionTime": "2023-02-22T17:45:24Z",
                "lastUpdateTime": "2023-02-22T17:45:24Z",
                "message": "ReplicaSet \"addresses-dd575cdc9\" has timed out progressing.",
                "reason": "ProgressDeadlineExceeded",
                "status": "False",
                "type": "Progressing"
            }
        ],
        "observedGeneration": 1,
        "replicas": 1,
        "unavailableReplicas": 1,
        "updatedReplicas": 1
    }
}

I need to extract all the name properties of the configMapRef

So I used thejq to extract like below:

cat file.json | jq '.spec.template.spec.containers[].envFrom[] | select(.configMapRef) '

And I get the below output:

{
  "configMapRef": {
    "name": "db-mssql-asd-connection"
  }
}
{
  "configMapRef": {
    "name": "db-postgre-asd-connection"
  }
}
{
  "configMapRef": {
    "name": "app-url"
  }
}

But I need to get the name so my output looks like this:

db-mssql-asd-connection
db-postgre-asd-connection
app-url

I tried many things such as;

// not wokring
jq '.spec.template.spec.containers[].envFrom[] | select(.configMapRef) | jq .name'

// not wokring
jq '.spec.template.spec.containers[].envFrom[] | select(.configMapRef.name) | jq .name'

And I want to store thee values as a bash array so I can iterate through them. Can someone please help me with this?

Please take a look at [How to create a **Minimal**, Complete, and Verifiable example](http://stackoverflow.com/help/mcve). — Cyrus, Feb 22 '23 at 18:24
Related: https://stackoverflow.com/questions/75538320/xidel-is-it-possible-to-retrieve-specific-nested-values-from-object — Gilles Quénot, Feb 22 '23 at 21:33

pmf · Answer 1 · 2023-02-22T19:24:34.360

The jq part

The way you are using select isn't productive. select emits either nothing (in which case that particular pipe branch dies), or its whole input (in which case you'd still need to descend into .configMapRef yourself). My suggestion is to unconditionally descend into it (and into .name as well), which may produce null values in the otherwise dying branches. You can filter them out either explicitly by using the alternative operator // with empty, or directly selecting either all non-null | values or, more constrainingly, only | strings. Either way, use the -r option to output raw text (instead of JSON):

jq -r '.spec.template.spec.containers[].envFrom[].configMapRef.name // empty' file.json

jq -r '.spec.template.spec.containers[].envFrom[].configMapRef.name | values' file.json

jq -r '.spec.template.spec.containers[].envFrom[].configMapRef.name | strings' file.json

db-mssql-asd-connection
db-postgre-asd-connection
app-url

The bash loop part

There are ways to escape the output (search for @sh) and to format it e.g. to fit to a declare statement (example), which will generate an indexed bash array. But in this case, it'd be simpler to just iterate over the lines output:

while read -r line
do … "$line" …
done < <(jq -r '…' file.json)

`parse error: Expected another key-value pair at line 7, column 9` — Gilles Quénot, Feb 22 '23 at 18:45
@GillesQuénot This is (obviously) a copy-paste error, not present in the original data, as OP themselves has no issues executing other jq filters. See "So I used the jq to extract like below […] And I get the below output". — pmf, Feb 22 '23 at 18:49

score 1 · Accepted Answer · answered Feb 22 '23 at 18:36

1

Just select the envFrom objects that has the property, then get the desired property in the final result.

$ jq -r '.spec.template.spec.containers[].envFrom[]
  | select(has("configMapRef")).configMapRef.name' file.json
db-mssql-asd-connection
db-postgre-asd-connection
app-url

answered Feb 22 '23 at 18:36

Jeff Mercado

129,526
32
251
272

`parse error: Expected another key-value pair at line 7, column 9` – Gilles Quénot Feb 22 '23 at 18:45
2

Yes... the example json is technically not valid json (excess trailing comma)... but the intent is obviously clear with a trivially fixable error. – Jeff Mercado Feb 22 '23 at 18:47
1

You should mention that – Gilles Quénot Feb 22 '23 at 18:48
If there was an actual structural error in the json and it was unclear what it would be intended, then the question would be unanswerable and we would have to wait for clarification. But again, the structure was obvious and any parser that wasn't as strict as jq wouldn't complain. There's no need for any of us to either since it doesn't change how we would approach the question or answers. – Jeff Mercado Feb 23 '23 at 03:23

Gilles Quénot · Answer 3 · 2023-02-22T21:35:03.107

With some Unix* pipes. Works also with your broken JSON, unlike the other answers

grep and regex are universal. gron makes the things sufficiently atomic to be able to use grep the reliable way.

For simple task, jq is the way to go, but if you need > 10 mn to find the good syntax, gron is a very nice tool to have in the toolchest.

The reason to use this, is that I refuse to go too deeper with jq. If you need to know another 'language' for each type of structured data files, it's a mess. I prefer simpler/shorter solutions. Here one:

$ yq . file.json | gron | grep -oP 'configMapRef\.name = "\K[^"]+' 
db-mssql-asd-connection
db-postgre-asd-connection
app-url

I use yq, because it's less strict than gron and jq:

$ jq . file.json
parse error: Expected another key-value pair at line 7, column 9

and

$ gron file.json
failed to form statements: invalid character '}' looking for beginning of object key string

If the json was better formatted, I was able to wrote:

gron 'https://gist.githubusercontent.com/sputnick-dev/7da057b951c182785dfb837194388350/raw/eaf064c6b44c8304e111fa4e6ee3638d47a4ef48/object.json' |
    grep -oP 'configMapRef\.name = "\K[^"]+'

Python yg works also with...
golang yq
gron Make JSON greppable!

Why use grep when jq (and yq) can do it? Processing json as raw text seems rather pointless when there are json processing tools available. — Jeff Mercado, Feb 22 '23 at 18:50
If you read carefully, `jq` give error here. And I explained my point of view with complicated 'language' like `jq`. For simple task, it's the way to go, but if you need > 10 mn to find the good syntax, `gron` is a very nice tool. And `grep` and `regex` are _universal_. `gron` makes the things sufficiently _atomic_ to be able to use `grep` the reliable way — Gilles Quénot, Feb 22 '23 at 18:53

ufopilot · Answer 4 · 2023-02-22T19:04:04.687

-1

$ jq -r '.spec.template.spec.containers[].envFrom[].configMapRef|select( . != null )|.name' file.json
db-mssql-asd-connection
db-postgre-asd-connection
app-url

Using AWK
$ awk -F'"' '/configMapRef/{getline; print $--NF}' <(json_pp <file.json) 
$ awk -F'"' '/configMapRef/{getline; print $--NF}' <(jq -r . file.json)
db-mssql-asd-connection
db-postgre-asd-connection
app-url

edited Feb 22 '23 at 19:04

answered Feb 22 '23 at 18:51

ufopilot

3,269
2
10
12

`parse error: Expected another key-value pair at line 7, column 9` – Gilles Quénot Feb 22 '23 at 18:58
And for `json_pp`: `unexpected end of string while parsing JSON string, at character offset 163 (before ",\n "creation...") at /home/me/perl5/perlbrew/perls/perl-5.36.0/bin/json_pp line 59.` – Gilles Quénot Feb 22 '23 at 18:58
The original file has an error on line 6 `,` at the end – ufopilot Feb 22 '23 at 19:00
@GillesQuénot it works.. Have no erros – ufopilot Feb 22 '23 at 19:02
@ufopilot You could [shorten](https://github.com/stedolan/jq/blob/master/src/builtin.jq#L63) `select( . != null )` to `values` (since jq 1.6) – pmf Feb 22 '23 at 19:29
You removed the trailing `,` then – Gilles Quénot Feb 22 '23 at 20:23
@GillesQuénot yes i did – ufopilot Feb 23 '23 at 08:58

jq - how to get all the values from a object within object

4 Answers4

The jq part

The bash loop part

With some Unix* pipes. Works also with your broken JSON, unlike the other answers

Linked