Im trying to use json to pull a variable into a loop

Question

I'm copying json? from two websites to try to achieve my goal

https://softwaredevelopmentstuff.com/2017/05/02/downloading-an-aws-glacier-archive-step-by-step/

https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5

This works:

#!/bin/bash

file='/home/nat/Documents/glacier_access/output.json'

if [[ -z ${AWS_ACCOUNT_ID} ]] || [[ -z ${AWS_REGION} ]] || [[ -z ${AWS_VAULT_NAME} ]]; then
    echo "Please set the following environment variables: "
    echo "AWS_ACCOUNT_ID"
    echo "AWS_REGION"
    echo "AWS_VAULT_NAME"
    exit 1
fi

archive_ids=$(jq .ArchiveList[].ArchiveId < $file)

for archive_id in ${archive_ids}; do
    echo "Downloading Archive: ${archive_id}"
     aws glacier delete-archive --archive-id=${archive_id} --vault-name ${AWS_VAULT_NAME} --account-id ${AWS_ACCOUNT_ID} --region ${AWS_REGION}
 
done

as does this:

# file to loop action on archives
#!/bin/bash

file='/home/nat/Documents/glacier_access/output.json'

if [[ -z ${AWS_ACCOUNT_ID} ]] || [[ -z ${AWS_REGION} ]] || [[ -z ${AWS_VAULT_NAME} ]]; then
    echo "Please set the following environment variables: "
    echo "AWS_ACCOUNT_ID"
    echo "AWS_REGION"
    echo "AWS_VAULT_NAME"
    exit 1
fi

archive_ids=$(jq .ArchiveList[].ArchiveId < $file)

for archive_id in ${archive_ids}; do
    echo "Downloading Archive: ${archive_id}"
aws glacier initiate-job \
    --account-id 510422384120\
    --vault-name ebony-backup \
    --job-parameters '{
        "Type": "archive-retrieval",
        "Description": "Download all my archives",
        "ArchiveId": "w0GUthLDLDR6NL1z4c53M1IFktxBCrW_qZ2Qm_",
    }'
 
done

but I cant reference the variable in the loop

so this doesnt work:

"ArchiveId": "${archive_id}"

nor this:

"ArchiveId": "{archive_id}"

An important thing to note is that the ArcihiveID MUST be referenced from inside the job parameters section.

Any help gratefully received. I'm assuming this is a JSON question?

@ikegami Thanks for the ping, I misread the problem statement. — tripleee, Jan 05 '22 at 07:01

ikegami · Accepted Answer · 2022-01-05T14:18:43.987

Your claim that you can't use the variable in the loop is false. You've proved this using echo "Downloading Archive: ${archive_id}". This means the problem is actually with the second command of the loop.

Specifically, the issue is your construction of the string used as the parameter to --job-parameters. Everything in single quotes is produced exactly, so you produce the string

{
        "Type": "archive-retrieval",
        "Description": "Download all my archives",
        "ArchiveId": "${archive_id}"
    }

Four solutions follow.

Creating the JSON from the shell using double-quoted string

    --job-parameters "{
        \"Type\":        \"archive-retrieval\",
        \"Description\": \"Download all my archives\",
        \"ArchiveId\":   \"$archive_id\"
    }"

In general, I hate the approach of generating code from a shell script. It is however safe in this case since the id consists entirely of word characters. We do end up with a lot of backslashes, though.

Creating the JSON from the shell using a here-doc

    --job-parameters <<EOS
{
   "Type":        "archive-retrieval",
   "Description": "Download all my archives",
   "ArchiveId":   "$archive_id"
}
EOS

This is still generating code from a shell script. But by using a here-doc, " is no longer special, so removed the need to escape them.

Creating the JSON using jq

    --job-parameters "$(
       jq -n --arg archive_id "$archive_id" '
          {
             "Type":        "archive-retrieval",
             "Description": "Download all my archives",
             "ArchiveId":   $archive_id
          }
       '
    )"

Note that this uses the jq variable $archive_id (created using --arg) in a jq program, so no quotes around it. jq will produce a correct JSON string literal for it (including it if contains special characters such as ").

This is a lot cleaner, and works even if archive_id contains special characters. It is a little slower, having to launch jq each time.

Creating the JSON using a single instance of jq

This addresses the issue of having to launch jq multiple times.

jq -c '
   .ArchiveList[] |
   {
      "Type":        "archive-retrieval",
      "Description": "Download all my archives",
      "ArchiveId":   .ArchiveId
   }
' "$file" |
while read -r job; do
   aws glacier initiate-job     \
      --account-id 510422384120 \
      --vault-name ebony-backup \
      --job-parameters "$job"
done

Note that there should not be a comma after the archive id. Trailing commas aren't allowed in JSON.

Im trying to use json to pull a variable into a loop

1 Answers1