Remove substring from a json retrieved in a bash script

Question

I do not have experience working with bash, and I have to update a small script to remove a substring from all the values in a JSON with a common pattern /development/dev/. I get something like this from GetParametersByPath (AWS service):

{
    "Parameters": [
        {
            "Name": "/development/dev/var1",
            "Type": "String",
            "Value": "Saanvi Sarkar",
            "Version": 1
        },
        {
            "Name": "/development/dev/var2",
            "Type": "String",
            "Value": "Zhang Wei",
            "Version": 1
        },
        {
            "Name": "/development/dev/var3",
            "Type": "String",
            "Value": "Alejandro Rosalez",
            "Version": 1
        },
        
    ]
}

I wanna remove the substring "/development/dev/" for all the Name values.

This is what I have at the moment

  // parameter_store_path has the value "/development/dev/"
 jq_actions=$(echo -e ".Parameters | .[] | [.Name, .Value] | \042\(.[0])=\(.[2])\042 | sub(\042${parameter_store_path}/\042; \042\042)")

// function that returns the JSON 
  aws ssm get-parameters-by-path \
  --path $parameter_store_path \
  --with-decryption \
  --region eu-west-2 \
  | jq -r "$jq_actions" >> /opt/elasticbeanstalk/deployment/custom_env_var

  cp /opt/elasticbeanstalk/deployment/custom_env_var /opt/elasticbeanstalk/deployment/env

  #Remove temporary working file.
  rm -f /opt/elasticbeanstalk/deployment/custom_env_var

  #Remove duplicate files upon deployment.
  rm -f /opt/elasticbeanstalk/deployment/*.bak

I was reading some docs about bash, and I saw that I can replace part of the string using ${FOO#prefix}, but I don't know how to implement it in this code.

tripleee · Accepted Answer · 2022-10-29T20:07:33.610

2

You already seem to know how to perform substitutions in jq. Just do slightly more.

aws ssm get-parameters-by-path \
  --path "$parameter_store_path" \
  --with-decryption \
  --region eu-west-2 |
jq -r '.Parameters | .[] |
  "\(.Name | sub("/development/dev/"; ""))=\(.Value)"'

(The subscript [2] didn't make sense here - I'm guessing you meant [1]. However, the subscripting seemed superfluous, anyway; I refactored to simplify it away, and apply the substitution only on the Name. Perhaps also add double quotes around the Value?)

Demo: https://jqplay.org/s/o7Exg4Ns6BU

If you want to pass in the path as a variable, try

jq --arg path "/development/dev/" \
   -r '.Parameters | .[] |
  "\(.Name | sub($path; ""))=\"\(.Value)\""'

Demo: https://replit.com/@tripleee/aws-jq-demo#main.sh

This also gets rid of the useless use of echo -e. In this particular case, I simply switched to single quotes instead of double (though notice that they do slightly different things - generally prefer single quotes, unless you need to perform command substitution or variable substitution in your string, or if it needs to contain literal single quotes. Bash also provides $'\047C-style\047 strings' with a backslash interpretation facility similar to the one in echo -e - the example demonstrates how to embed literal single quotes with the octal escape \047).

A parameter expansion only really makes sense when what you have is already a string in a Bash variable.

Here's a refactoring of your script which also avoids the weird cp + rm in favor of a simple mv. I'm guessing you are writing something else to the target file earlier in the script and that's why you append these values from jq.

#!/bin/bash
parameter_store_path="/development/dev/"

aws ssm get-parameters-by-path \
  --path "$parameter_store_path" \
  --with-decryption \
  --region eu-west-2 |
jq --arg path "$parameter_store_path" \
   -r '.Parameters | .[] |
  "\(.Name | sub($path; ""))=\"\(.Value)\""
  ' >> /opt/elasticbeanstalk/deployment/custom_env_var

mv /opt/elasticbeanstalk/deployment/custom_env_var /opt/elasticbeanstalk/deployment/env

rm -f /opt/elasticbeanstalk/deployment/*.bak

edited Oct 29 '22 at 20:07

answered Oct 29 '22 at 10:16

tripleee

175,061
34
275
318

I think your solution is quite close, but the result is not 100% correct, because the env variables file should not has "" var1=Saanvi Sarkar (this is correct). About this "You already seem to know how to perform substitutions in jq. Just do slightly more" Not too much, I just learned today, I am learning bash. Thanks for your help. – R0bertinski Oct 29 '22 at 10:55
You did not explain what you expect; but the fix is easy - I updated to add the `-r` option like in your original. (The output without any quoting is slightly jarring; it's neither valid Bash nor valid JSON.) – tripleee Oct 29 '22 at 11:42
The recipe to add double quotes around the value is just a quick hack, and won't work correctly if the actual value contains literal double quotes. – tripleee Oct 29 '22 at 11:58
The values never will have quotes, they are sanitized when they are added in the SSM parameter store in AWS , then this is not a thing to be worry I guess. Thanks you very much , i will check when back to the computer. – R0bertinski Oct 29 '22 at 12:10
Well something still not correct, I am trying to add your code to the script but possible I am doing something wong. jq_actions=$(echo -e ".Parameters | .[] | [.Name, .Value] | sub(\042${parameter_store_path}/\042; \042\042)") I got this error: "jq: error (at :1030): array (["/development/dev) cannot be matched, as it is not a string". I do not know why the error says "array" . – R0bertinski Oct 29 '22 at 14:49
I continue to strongly recommend avoiding that `echo -e` antipattern. – tripleee Oct 29 '22 at 16:02
If you want to pass in a variable to `jq`, use the `--arg` option. – tripleee Oct 29 '22 at 16:04
That's not my code, anyway. It looks like redux of your earlier attempt. Basically it's telling you that you forgot to quote the first argument to `sub()` … I think, but we have to guess and read between the lines a bit. – tripleee Oct 29 '22 at 16:12
I updated the answer to show exactly how to use `--arg`. But going forward, please don't move the goalposts. Accept an answer which solves the problem you actually asked about; if you then have a new problem and can't solve it yourself, ask a new question. – tripleee Oct 29 '22 at 20:09
I am using your example (thanks for that), and I changed it a bit to get the strings without " " to be valid as env variables in my env file (I cannot add them between " "). I was able to remove the first " at the beginning of the values, but not at the end, any idea? thanks. jq --arg path "/development/dev/" \ -r '.Parameters | .[] | "\(.Name | sub($path; ""))=\(.Value | sub(" "; ""))\""' <<\: This is the current result var1=SaanviSarkar" var2=ZhangWei" var3=AlejandroRosalez" – R0bertinski Oct 30 '22 at 03:28
The first example shows how to omit the quotes. – tripleee Oct 30 '22 at 05:45
Yeah your right, many thanks. Definitely, I have to learn bash, not choice :) – R0bertinski Oct 30 '22 at 06:09
Maybe also notice that the JQplay links have an automatically generated explanation of the script near the bottom of the page. – tripleee Oct 30 '22 at 07:22

Remove substring from a json retrieved in a bash script

1 Answers1

Linked