How to find and replace json with shell variables using jq?

Question

I have read properties with jq from a json object and have stored them to variables.

I want to now read these variables and essentially find and replace a word inside the string with a global shell variable.

I've set my json ID's from my JSON file

# Set Json ID's
TARGET_ID=$(jq '.DefaultCacheBehavior.TargetOriginId' distconfig.json)
DOMAIN_NAME=$(jq '.Origins.Items[0].DomainName' distconfig.json)
ORIGIN_ID=$(jq '.Origins.Items[0].Id' distconfig.json)

echo "$TARGET_ID"
echo "$DOMAIN_NAME"
echo "$ORIGIN_ID"

This returns

"S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com"
"stag4.example.io.s3-website.us-east-2.amazonaws.com"
"S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com"

I have my location id variable and would like to write it to find and replace all stag4 references in those 3 ID's.

Then I would like to write those 3 ID's to the initial json object, or create a temp version of it.

Example, if: $DOMAIN_NAME is"stag4.example.io.s3-website.us-east-2.amazonaws.com"

I would like to essentially have it set to:

$LOCATION_NAME="stag6"
DOMAIN_LOCATION="example.io"

"$DOMAIN_NAME=S3-Website-\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
"$TARGET_ID=\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
"$ORIGIN_ID=S3-Website-\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"

Then write those 3 to the temp or new json file so I can run my cloudformation command:

aws cloudfront create-distribution --distribution-config file://disttemp.json

I have now built out the proper variables from the initial json file like so:

$LOCATION_NAME="stag6"
DOMAIN_LOCATION="example.io"

echo "Build New IDs"
TARGET_ID_BUILT="S3-Website-$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
DOMAIN_NAME_BUILT="$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
ORIGIN_ID_BUILT="S3-Website-$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"

echo "$TARGET_ID_BUILT"
echo "$DOMAIN_NAME_BUILT"
echo "$ORIGIN_ID_BUILT"

How do I write these variables to the json file with jq?

EDIT: Sample of distconfig.json requested – domain/creds swapped to example

{
  "CallerReference": "my-test-distribution-2",
  "Comment": "", 
  "CacheBehaviors": {
      "Quantity": 0
  }, 
  "IsIPV6Enabled": true, 
  "Logging": {
      "Bucket": "", 
      "Prefix": "", 
      "Enabled": false, 
      "IncludeCookies": false
  }, 
  "WebACLId": "", 
  "Origins": {
      "Items": [
          {
              "OriginPath": "", 
              "CustomOriginConfig": {
                  "OriginSslProtocols": {
                      "Items": [
                          "TLSv1", 
                          "TLSv1.1", 
                          "TLSv1.2"
                      ], 
                      "Quantity": 3
                  }, 
                  "OriginProtocolPolicy": "http-only", 
                  "OriginReadTimeout": 30, 
                  "HTTPPort": 80, 
                  "HTTPSPort": 443, 
                  "OriginKeepaliveTimeout": 5
              }, 
              "CustomHeaders": {
                  "Quantity": 0
              }, 
              "Id": "S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com", 
              "DomainName": "stag4.example.io.s3-website.us-east-2.amazonaws.com"
          }
      ], 
      "Quantity": 1
  },
}
"DefaultRootObject": "", 
  "PriceClass": "PriceClass_All", 
  "Enabled": true, 
  "DefaultCacheBehavior": {
      "TrustedSigners": {
          "Enabled": false, 
          "Quantity": 0
      }, 
      "LambdaFunctionAssociations": {
          "Quantity": 0
      }, 
      "TargetOriginId": "S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com", 
      "ViewerProtocolPolicy": "redirect-to-https", 
      "ForwardedValues": {
          "Headers": {
              "Quantity": 0
          }, 
          "Cookies": {
              "Forward": "none"
          }, 
          "QueryStringCacheKeys": {
              "Quantity": 0
          }, 
          "QueryString": false
      }, 
      "MaxTTL": 31536000, 
      "SmoothStreaming": false, 
      "DefaultTTL": 86400, 
      "AllowedMethods": {
          "Items": [
              "HEAD", 
              "GET"
          ], 
          "CachedMethods": {
              "Items": [
                  "HEAD", 
                  "GET"
              ], 
              "Quantity": 2
          }, 
          "Quantity": 2
      }, 
      "MinTTL": 0, 
      "Compress": true
  }, 
  "ViewerCertificate": {
      "SSLSupportMethod": "sni-only", 
      "ACMCertificateArn": "xxxx", 
      "MinimumProtocolVersion": "TLSv1.1_2016", 
      "Certificate": "xxxx", 
      "CertificateSource": "acm"
  }, 
  "CustomErrorResponses": {
      "Quantity": 0
  }, 
  "HttpVersion": "http2", 
  "Restrictions": {
      "GeoRestriction": {
          "RestrictionType": "none", 
          "Quantity": 0
      }
  }, 
  "Aliases": {
      "Quantity": 0
  }
}

This link might help https://stackoverflow.com/questions/42716734/modify-a-key-value-in-a-json-using-jq — BlackPearl, Apr 20 '19 at 06:15
Yeah i'm trying to use a temp file like suggested im just not sure how to do the 'find and replace' action of setting my new variables in the new file in place of where I got the old variables in the initial file — h0bb5, Apr 20 '19 at 06:29
@user3648969 see [How to create a Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve). the sample you provided is an incomplete JSON value, and it doesn't have a key named `DefaultCacheBehavior` whereas your code indicates that `TARGET_ID` is retrieved from there. — oguz ismail, Apr 20 '19 at 12:17
@oguzismail Sorrry, I put an ellipsis at the bottom of the object to indicate it continued, I just didn't feel like cluttering up the post with the full object. As long as I know the function to find and replace one of the variables I can apply it to the different paths — h0bb5, Apr 20 '19 at 17:41

score 0 · Answer 1 · answered May 14 '19 at 16:08

You should use sed to do the substitution and then inject the value back into the JSON.

echo $TARGET_ID | sed 's/stag4/stag5/g'

Outputs S3-Website-stag5.example.io.s3-website.us-east-2.amazonaws.com

Next we'll put the value back into the original JSON, this will technically output a new JSON and does not edit the file, however, you can easily solve for this on the output by temporarily saving to a tmp file.

We will use the --arg flag to reference our bash variable and set the new value for our field

cat distconfig.json | jq --arg TARGET_ID $TARGET_ID '.DefaultCacheBehavior.TargetOriginId = $TARGET_ID' > tmp.json && mv tmp.json distconfig.json

How to find and replace json with shell variables using jq?

1 Answers1