How to iterate over JSON array with jq?

Question

I'm building a script to download all CodeCommit repositories at once.

REPOS=$(aws codecommit list-repositories)

echo $REPOS | jq -r '.[]' | while read name ; do
    git clone XXX
done

In first line I get all repos JSON like this:

[
  {
    "repositoryName": "repo_a",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_b",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_c",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_d",
    "repositoryId": "XXXXXX"
  }
]

I need simple iterate this json, to get attributes repositoryName and execute git clone for each repository.

But in this example the command jq -r '.[]' don't works.... This return the entire json on each repeat.

Sounds like you want `.[][]` to go down an additional nesting layer. — Charles Duffy, Jun 24 '21 at 18:44
To be clear, we can't reproduce your bug unless you give us the original output from `aws codecommit list-repositories`. A better [mre] would, instead of running a tool that only you can run and get your output, maybe hardcode sample data that's reflective of the real format (so f/e, you can define a `aws` function that writes data in the same format to its stdout). — Charles Duffy, Jun 24 '21 at 18:44
Also, `echo $REPOS |` is itself buggy. See [I just assigned a variable, but `echo $variable` prints something else](https://stackoverflow.com/questions/29378566/i-just-assigned-a-variable-but-echo-variable-shows-something-else) -- you should _always_ quote your expansions, as in, `echo "$REPOS" | ...` — Charles Duffy, Jun 24 '21 at 18:46

Cyrus · Answer 1 · 2021-06-24T19:04:16.323

17

echo "$REPOS" | jq '.[].repositoryName' | while read -r repo; do echo "do something with $repo"; done

Output:

do something with "repo_a"
do something with "repo_b"
do something with "repo_c"
do something with "repo_d"

Or without quotes:

echo "$REPOS" | jq -r '.[].repositoryName' | while read -r repo; do echo "do something with $repo"; done

Output:

do something with repo_a
do something with repo_b
do something with repo_c
do something with repo_d

edited Jun 24 '21 at 19:04

answered Jun 24 '21 at 18:47

Cyrus

84,225
14
89
153

`xargs -L1` would be useful too. – glenn jackman Jun 24 '21 at 18:48
@glennjackman: Yes, I don't know exactly what else he's up to. – Cyrus Jun 24 '21 at 18:49

score 3 · Answer 2 · answered Jun 24 '21 at 19:04

I found the problem..... this command:

REPOS=$(aws codecommit list-repositories)

Returns a json like this:

{
  "repositories": [
  {
    "repositoryName": "repo_a",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_b",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_c",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_d",
    "repositoryId": "XXXXXX"
  }
]
}

I wasn't considering the 'repositories' field in jq.....

This script works fine now:

REPOS=$(aws codecommit list-repositories)

echo $REPOS | jq -r '.repositories[].repositoryName' | while read name ; do
    git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/$name
done

Thank you for support. Have a good day :)

score 1 · Answer 3 · answered Jun 12 '23 at 21:00

If you want to iterate over JSON values instead of a single field value, you can use the -c option so you end up with one JSON element per line:

repositories='{
  "repositories": [
  {
    "repositoryName": "repo_a",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_b",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_c",
    "repositoryId": "XXXXXX"
  },
  {
    "repositoryName": "repo_d",
    "repositoryId": "XXXXXX"
  }
]
}'

while read repository
do
  name=$(echo "$repository" | jq -r .repositoryName)
  id=$(echo "$repository" | jq -r .repositoryId)

  echo "${id}=${name}"
done < <(echo "$repositories" | jq -c '.repositories[]')

How to iterate over JSON array with jq?

3 Answers3