for loop returns only first array element in bash script

Question

I want to iterate over the below array received from a curl response and print the values of all the "path" properties on a new line but I am getting unwanted results everytime (may be i dont know how to do)

{
"values": [
    {
        "old": {
            "path": "textfile.txt"
        },
        "size":1
    },
    {
        "old": {
            "path": "text1file.txt"
        },
        "size":1
    }
]
}

Below is my bash script which returns only the first value, also when i try to get the length it always returns 1 instead of 2

FILES=$(curl "https://url" | jq '(.values[].old.path)')
echo "$FILES"
echo "length is: ${#FILES[@]}"
for i in "${FILES[@]}"; do
    echo "num1 ${i}"
done

The response received for line 2 is below:

"textfile.txt"
"text1file.txt"

The response received for line 3:

length is: 1

The expected response of line 5 is:

num1 "textfile.txt"
num1 "text1file.txt"

But instead it gives:

num1 "textfile.txt"
     "text1file.txt"

PS: I am doing the above script in bitbucket pipelines to implement CD process.

Does any one know what's wrong in the above for loop?

EDIT: I also tried the below script it still gives only the first value.

  curl -H 'Content-Type:application/json' https://url >> myfilenames.json
  for i in $(jq -r '.values[].old.path' myfilenames.json); do
      echo "${i}"
  done

`FILES=$(curl "https://url" | jq '(.values[].old.path)')` is _not_ an array assignment, but assignment to a variable — Inian, Aug 24 '20 at 16:32
I edited the new way of loop I learnt from some of the stackoverflow answers, it doesnt work as well. Any idea how to fix this? — nz_pree, Aug 24 '20 at 16:47
@nz_pree, what "new way of loop" are you talking about? The `for item in "${array[@]}"` approach works only if your value is truly an array. `files=$(curl ... | jq ...)` does not create an array, so you can't use `"${files[@]}"` with it. Use the code in the answer by Shawn to create a true array, and _then_ `"${files[@]}"` will work properly. — Charles Duffy, Aug 24 '20 at 17:26

Shawn · Accepted Answer · 2020-08-24T17:29:03.593

4

If you want to save the paths in a bash array, you need something like:

mapfile -t files < <(curl ... | jq '.values[].old.path')
echo "Length is ${#files[@]}"
for file in "${files[@]}"; do
    echo "num1 $file"
done

(See Correct Bash and shell script variable capitalization)

Just in case you don't want the double quotes around the path even though they're present in your question, use jq -r.

If you don't want the quotes and the paths might have newlines in them (and you have a development version of jq installed):

mapfile -d '' -t files < <(curl ... | jq -0 '.values[].old.path')

The -0 option is not (yet) in an official release. So this version will be more useful:

mapfile -d '' -t files < <(curl ... | jq -j '.values[] | "\(.old.path)\u0000"')

If you're using a version of bash older than 4.4, see Bash FAQ 005 for a workaround using read in a loop instead of mapfile.

edited Aug 24 '20 at 17:29

answered Aug 24 '20 at 16:59

Shawn

47,241
3
26
60

This is really no better than just reading the output of jq and iterating, and does not address the problem of newlines in the paths. IOW, there's no point in putting it in an array if you do this. – William Pursell Aug 24 '20 at 17:07
@WilliamPursell Er, I included an example that handles newlines in raw strings. Though admittedly it requires a development version of `jq` as 1.6 doesn't support the `-0` option. And just iterating doesn't give you what the OP wants with first printing out the number of records and then the records themselves. – Shawn Aug 24 '20 at 17:10
(Without `-r/-0/-j`, `jq` will output newlines in strings as `\n` escapes, not actual newlines) – Shawn Aug 24 '20 at 17:21
@WilliamPursell, _if combined with_ iterating over the array's contents properly, this is definitely much better practice than what the OP was doing earlier, as any of the NUL-based variants will work properly with all possible filenames. That said, I do agree that to be fully complete this needs to show how to iterate over the array (`for file in "${files[@]}"; do ...`). – Charles Duffy Aug 24 '20 at 17:24
@CharlesDuffy OP already seems to know how to do that part so I left it out. It's just getting values into an array that they were having problems with. – Shawn Aug 24 '20 at 17:27
Rereading the question, I agree that they know how to do it; I'm not sure they know _when it's appropriate_, as they've experimented with other mechanisms as well. – Charles Duffy Aug 24 '20 at 17:27
@Shawn Thanks alot for your answer. I tried the script suggested by you as below: - mapfile -t files << (curl "https://url" | jq '.values[].old.path') - echo "Length is ${#files[@]}" - for file in "${files[@]}"; do - echo "num1:${file}" - done But it givesthis : bash: /opt/atlassian/pipelines/agent/tmp/bashScript7963299335666580030.sh: line 17: syntax error near unexpected token `(' – nz_pree Aug 24 '20 at 17:43
@nz_pree I think you removed a space where there should be one and added a space where there shouldn't be one. – Shawn Aug 24 '20 at 17:52
@Shawn yes i corrected the space thing and it gave correct result for length(i.e, 2) but it printed only the first filename – nz_pree Aug 24 '20 at 17:58
@nz_pree Works for me. – Shawn Aug 24 '20 at 18:19

for loop returns only first array element in bash script

1 Answers1