Add to existing json file using jq

Question

I have an Artifactory AQL Spec file in JSON format. The spec file is as follows:

{
  "files": [
    {
      "aql": {
        "items.find": {
          "repo": "release-repo",
          "modified": { "$before": "30d" },
          "type": { "$eq": "folder" },
          "depth": "2"
        }
      }
    }
  ]
}

let's say I run a gitlab api query to acquire a list of SHAs that I want to iterate through and add to this json spec file.. The list of SHAs are assigned to a variable..

"a991fef6bb9e9759d513fd4b277fe3674b44e4f4"
"5a562d34bb1d4ab4264acc2c61327651218524ad"
"d4e296c35644743e58aed35d1afb87e34d6c8823"

I would like to iterate through all these commit IDs in and add them one by one to the json so that they are in this format:

{
  "files": [
    {
      "aql": {
        "items.find": {
          "repo": "release-repo",
          "modified": { "$before": "30d" },
          "type": { "$eq": "folder" },
          "$or": [
            {
              "$and": [
                {
                  "name": {
                    "$nmatch": "*a991fef6bb9e9759d513fd4b277fe3674b44e4f4*"
                  }
                }
              ]
            },
            {
              "$and": [
                {
                  "name": {
                    "$nmatch": "*5a562d34bb1d4ab4264acc2c61327651218524ad*"
                  }
                }
              ]
            },
            {
              "$and": [
                {
                  "name": {
                    "$nmatch": "*d4e296c35644743e58aed35d1afb87e34d6c8823*"
                  }
                }
              ]
            }
          ],
          "depth": "2"
        }
      }
    }
  ]
}

The list of SHAs returned from the gitlab api query will be different everything and that's why I'd like this to be a dynamic entry or update every time. The number of returned SHAs will also be different... Could return 10 one day or it could return 50 on another day.

You have quotes in your shell variable? (What *exactly* is its format?) — Charles Duffy, Feb 08 '18 at 17:11
@CharlesDuffy the variable as follows: export result=$(curl --header "PRIVATE-TOKEN: *************" 'https://gitlab.com/api/v4/projects/265/repository/tags?per_page=800&order_by=updated&sort=desc' | jq '.[] | select (.name | contains ("release")) | .commit.id') Which returns: echo $results "a991fef6bb9e9759d513fd4b277fe3674b44e4f4" "5a562d34bb1d4ab4264acc2c61327651218524ad" "d4e296c35644743e58aed35d1afb87e34d6c8823" — Ebrahim Moshaya, Feb 08 '18 at 17:13
(I'm actually in the middle of an answer that assumes that it's a shell array, and that the quotes are syntactic rather than literal). — Charles Duffy, Feb 08 '18 at 17:13
@CharlesDuffy I can update the command to remove the quotes from the list of SHAs — Ebrahim Moshaya, Feb 08 '18 at 17:15
`echo $results` is much less useful than you'd think (its output is ambiguous in a surprising number of ways). Use `declare -p results` instead. — Charles Duffy, Feb 08 '18 at 17:17
([BashPitfalls #14](http://mywiki.wooledge.org/BashPitfalls#echo_.24foo) starts to touch on the faults of unquoted `echo`s... though even quoted, there's a lot of ambiguity; see the APPLICATION USAGE section of the [POSIX spec for `echo`](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html)). — Charles Duffy, Feb 08 '18 at 17:29
BTW, I don't recommend `export`ing variables unless you have a specific and compelling reason to do so. Exporting a shell variable into the environment means it's stored in the same region of memory also used for command-line arguments, so you're making your maximum allowed command-line length shorter. See for example https://stackoverflow.com/questions/48486201/bashshell-argument-too-long-message-shows-after-for-loop-echo-command/48523766#48523766 — Charles Duffy, Feb 08 '18 at 17:41

score 2 · Accepted Answer · answered Feb 08 '18 at 17:25

#!/usr/bin/env bash

template='{
  "files": [
    {
      "aql": {
        "items.find": {
          "repo": "release-repo",
          "modified": { "$before": "30d" },
          "type": { "$eq": "folder" },
          "$or": [],
          "depth": "2"
        }
      }
    }
  ]
}'

shas=(
  "a991fef6bb9e9759d513fd4b277fe3674b44e4f4"
  "5a562d34bb1d4ab4264acc2c61327651218524ad"
  "d4e296c35644743e58aed35d1afb87e34d6c8823"
)

jq -n \
        --argjson template "$template" \
        --arg shas_str "${shas[*]}" \
'
reduce ($shas_str | split(" ") | .[]) as $sha ($template;
  .files[0].aql["items.find"]["$or"] += [{
    "$and": [{"name": {"$nmatch": ("*" + $sha + "*")}}]
  }]
)
'

...emits as output:

{
  "files": [
    {
      "aql": {
        "items.find": {
          "repo": "release-repo",
          "modified": {
            "$before": "30d"
          },
          "type": {
            "$eq": "folder"
          },
          "$or": [
            {
              "$and": [
                {
                  "name": {
                    "$nmatch": "*a991fef6bb9e9759d513fd4b277fe3674b44e4f4*"
                  }
                }
              ]
            },
            {
              "$and": [
                {
                  "name": {
                    "$nmatch": "*5a562d34bb1d4ab4264acc2c61327651218524ad*"
                  }
                }
              ]
            },
            {
              "$and": [
                {
                  "name": {
                    "$nmatch": "*d4e296c35644743e58aed35d1afb87e34d6c8823*"
                  }
                }
              ]
            }
          ],
          "depth": "2"
        }
      }
    }
  ]
}

score 1 · Answer 2 · answered Feb 08 '18 at 19:05

Here is a reduce-free solution. It makes some inessential assumptions - that the sha strings are presented as a stream of strings on STDIN, and that the Artifactory spec is in a file named spec.json. Here is the jq program:

map( {"$and": [ {name: { "$nmatch": "*\(.)*" }}]} ) as $x
| $spec[0] | (.files[0].aql."items.find"."$or" = $x)

The jq invocation might look like this:

jq -s --slurpfile spec spec.json -f program.jq <<< "${shas[*]}"

Add to existing json file using jq

2 Answers2