Parsing string with escape characters into JSON and to Variable in Bash

Question

The Problem:

I am trying to fetch credentials from AWS Secrets Manager in my terminal, however the Keys and Values I want needs to be in JSON, however they come with a lot of escape characters due to quotes.

The Scenario:

After I fire the aws secretsmanager get-secret-value --secret-id snowflake-access-uat command, I get the credentials as below:

{
    "ARN": "arn:aws:secretsmanager:ap-regionnm-1:111111111111:secret:my-secret",
    "Name": "snowflake-access-uat",
    "VersionId": "dont-care",
    "SecretString": "{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": 1627104812.142
}

However, I am interested in Secret String only, for which I fire aws secretsmanager get-secret-value --secret-id snowflake-programmatic-access-uat | jq '.SecretString' command and receive this:

"{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}"

But since it has multiple escape characters, I am unable to leverage it with jq tree. I tried to get from this link for reference but I'm unable to make it work. Besides, I need the Keys and Values to be variables in my bash session.

NOTE: I cannot use any third party tools, since I need to automate this on CodeBuild (Run time fresh instance will be selected)

You're already showing and telling us that `jq` is enabled, so we're in a good place re: "third-party tools". — Charles Duffy, Oct 14 '21 at 13:10
While the linked duplicate's title asks only about double quotes, requesting raw output (as the accepted answer instructs) removes the other unwanted escaping as well. — Charles Duffy, Oct 14 '21 at 13:14
@CharlesDuffy Yes Charles, CodeBuild recently added Ubuntu series, so ya it does have jq but I am not well versed if how can I use Python in writing the buildspec.yml with it. — T3J45, Oct 14 '21 at 13:15
So, the thing to remember about YAML is that _all JSON is valid YAML_ — Charles Duffy, Oct 14 '21 at 13:15
...so you can write a JSON file with the same data, name it `buildspec.yml`, and you're set. — Charles Duffy, Oct 14 '21 at 13:16
(if you want to know what a given chunk of YAML looks like as JSON, the easiest way to find out is to write your sample YAML file on a system where you _can_ install new software; install a Python YAML library, of which there are many; load it into a Python variable with that library, and then use the built-in Python JSON library to save it to JSON). — Charles Duffy, Oct 14 '21 at 13:17
@CharlesDuffy the data in JSON is not a simple text, it is credentials instead. And Secrets Manager will be configured to rotate the password in Prod environment, so it wouldn't be a good practice. — T3J45, Oct 14 '21 at 13:18
Huh? You asked me "how I can use Python in writing the buildspec.yml with it", and I answered your question. If dynamically rewriting the buildspec.yml is a bad idea, then why did you _ask_ that question? — Charles Duffy, Oct 14 '21 at 13:18
@CharlesDuffy No no buddy, I think there is a misunderstanding. The point is I am supposed to get latest password from Secrets Manager which keeps on changing periodically, I just need to read that json and create it as variables for next statements to use it. I did not understand the point where you asked me to try things with Python. — T3J45, Oct 14 '21 at 13:50

Charles Duffy · Accepted Answer · 2021-10-14T13:17:39.217

The escape characters are there because you don't use -r with jq '.SecretString'. Change it to jq -r '.SecretString' and your output will instead be:

{"sf-user":"USER_123_ADMIN","sf-password":"FooBaarPassword","sf-db":"MY_SPL_DB","wh_name":"JOB_EXECUTOR","sf-role":"JOB_EXECUTOR_ROLE","sf-account":"icy-party"}

...which, being valid JSON, you can feed back into jq -r to retrieve individual fields.

SecretStringJson=$(... | jq -r '.SecretString')
### one jq call per field isn't the most efficient possible way but it's easy
sfUser=$(jq -r '.["sf-user"]' <<<"$SecretStringJson")
sfDb=$(jq -r '.["sf-db"]' <<<"$SecretStringJson")
# ...etc

Parsing string with escape characters into JSON and to Variable in Bash

The Problem:

The Scenario:

1 Answers1