0

I am trying to use a nested loop so that I can get the urls I desire, while also renaming the downloaded files to be unique. I am not getting something right.

The outcome is that I have unique file names (exactly like I want), but then they all have the same content, not at all what I want.

Here is what I have:

for image_name in `cat $IMAGE_TEST`;
do
  for manifest_url in `cat $MANIFEST_URL_LIST`;
  do
     curl -H "Authorization: Bearer $KC_ACCESS_TOKEN" $manifest_url > \
     manifests/$image_name.json
  done 
done

I have also tried:

for image_name in `cat $IMAGE_TEST`;
do
  while read -r manifest_url
  do
    curl -H "Authorization: Bearer $KC_ACCESS_TOKEN" $manifest_url > \
    mainifests/$image_name.json
  done < $MANIFEST_URL_LIST
done

and other combinations of this, but seem to be not seeing what I am missing. Please help!

swarr35
  • 89
  • 1
  • 8
  • Think about what your code does -- it's overwriting every `$image_name.json` with *every possible url*, so only the last URL is saved. – Charles Duffy Feb 13 '20 at 16:03
  • I'd guess you want the first image URL to be saved in the first filename, the second URL in the second filename, etc; we have Q&A entries on how to do that, but (in *any* language, not just bash) a nested loop is the wrong tool for that job altogether. – Charles Duffy Feb 13 '20 at 16:03
  • BTW, `for var in $(anything)` and its backtick-based alternative are generally bad practice -- see [BashPitfalls](http://mywiki.wooledge.org/BashPitfalls) entry #1. Use arrays instead: `read -t image_names <"$IMAGE_TEST"; read -t manifest_urls <"$MANIFEST_URL_LIST"; for idx in "${!image_names[@]}"; do curl "${manifest_urls[$idx]}" >"manifests/${image_names[$idx]}.json"; done` – Charles Duffy Feb 13 '20 at 16:05
  • (for other cases where you're looping over file input [BashFAQ #1](http://mywiki.wooledge.org/BashFAQ/001) practices are appropriate here, but that doesn't cover the two-input-sources-in-lockstep case). – Charles Duffy Feb 13 '20 at 16:06
  • BTW, another approach is `while IFS= read -r image_name <&3 && read -r manifest_url <&4; do curl "$manifest_url" >"manifests/$image_name.json"; done 3<"$IMAGE_TEST" 4<"$MANIFEST_URL_LIST"`, which is very much in the vein of BashFAQ #1. Either way, if you only want to use each filename once (and each url once), a nested loop is entirely the wrong tool for the job. – Charles Duffy Feb 13 '20 at 16:07
  • @CharlesDuffy what does `${!image_names[@]}` do? – swarr35 Feb 13 '20 at 16:09
  • Instead of iterating over the *content* of the image_names array (as `"${image_names[@]}"` does), it iterates over the *indexes* into that array. See https://wiki.bash-hackers.org/syntax/arrays – Charles Duffy Feb 13 '20 at 16:09
  • The while loops with two different files suggeted by Charles is a very cool approach. I'm going to have to play with that. My approach ( before the question got closed ) was to use a whiile loop as well. `paste "$IMAGE_TEST" "$MANIFEST_URL_LIST" | while read image_name manifest_url ; do ...` – Mark Feb 13 '20 at 16:11
  • @Mark, maybe add that as an answer on the linked duplicate? I don't see it covered there already. :) – Charles Duffy Feb 13 '20 at 16:11
  • @Mark, ...though it looks like that technique is already covered in https://stackoverflow.com/questions/56544497/nested-loop-for-reading-two-files-with-bash, which is arguably a better duplicate than the first one. – Charles Duffy Feb 13 '20 at 16:14
  • ```while read -t image_names <"$IMAGE_TEST"; do while read -t manifest_urls <"$MANIFEST_URL_LIST"; for idx in "${!image_names[@]}"; do curl -H "Authorization: Bearer $KC_ACCESS_TOKEN" "${manifest_urls[$idx]}" >"manifests/${image_names[$idx]}.json"; done done ``` What am I missing? – swarr35 Feb 13 '20 at 16:18
  • You're combining two separate alternatives there -- *either* use `while read` (with `-r`, not `-t`), *or* use `readarray`. The `"${arrayname[@]}"` syntax only makes sense if you're using `readarray`. Which is to say -- follow one or the other suggestions I made, but you can't do both of them at once. :) – Charles Duffy Feb 13 '20 at 16:20
  • (if you're targeting bash 3.x instead of a modern 4.x or 5.x release, that means you don't have `readarray`, so use the `while read` approach -- exactly as it was given above, using `-r`, and not using array syntax inside the loop). – Charles Duffy Feb 13 '20 at 16:22
  • @CharlesDuffy It worked, thanks so much for your time! I have much to learn! Any good bash resources beyond the sites you shared? For learning some more fundamentals? – swarr35 Feb 13 '20 at 16:32
  • 1
    The sites I linked to already -- the [bash-hackers' wiki](http://wiki.bash-hackers.org/) and the Wooledge wiki -- are the two resources I most recommend other than the [official manual itself](https://www.gnu.org/software/bash/manual/bash.html). In addition to the previously-linked BashPitfalls and BashFAQ pages on the Wooledge wiki, see also the [BashGuide](https://mywiki.wooledge.org/BashGuide). – Charles Duffy Feb 13 '20 at 16:39
  • 1
    @CharlesDuffy - I agree 100% - the duplicate you reference (stackoverflow.com/questions/56544497/…, ) is nearly a perfect fit for this question. – Mark Feb 13 '20 at 16:47

0 Answers0