1

I'm trying to loop through the contents of 3 files using a while loop in bash. They contain a list of names, aws accounts and aws account numbers.

But the loop isn't correct and it keeps repeating the first name from the list and the first aws environment from the list.

This is the output I see:

AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-lab

These are the files I'm trying to read from:

aws_users_all="source_files/aws_users_all.txt"

Sample output: 

aadatiya
abigailcharles 
tdunphy
broberts

Next file: 

aws_env_list="source_files/aws_environments_all.txt"

Sample output:

company-lab
company-stage
company-nonprod
company-prod

Last file: aws_account_numbers="source_files/aws_account_numbers.txt" Sample output:

  123456789191
  987654321211
  456721231213
  123213512321

And this is the code with the incorrect loop:

ofile=source_files/aws_access_keys/company-aws-access-keys-all-accounts.csv
while IFS= read -r aws_user_name
  do 
    while IFS= read -r aws_key
    do
      while IFS= read -r aws_account_num
      do
        user_lives_here=$(aws iam get-user --user-name "$aws_user_name" --profile="$aws_key" 2> /dev/null  | jq -r '.User.UserName')
        if [[ -z "$user_lives_here" ]]; then
          printf "AWS user name: %s does not exist in AWS account: %s\\n(%s)" "$aws_user_name" "$aws_key" "$aws_account_num"
        else
          echo "$aws_user_name,$user_access_key1,$key1_date_created,$key1_last_used,$key1AgeDays,$aws_key,$aws_account_num" >> $ofile
      fi
      done < "$aws_account_numbers"
    done < "$aws_env_list"
  done < "$aws_users_all"

If I take out one level (the account numbers level) the script behaves as expected and produces this output:

AWS user name: aadatiya does not exist in AWS account: company-lab
AWS user name: aadatiya does not exist in AWS account: company-bill
AWS user name: aadatiya does not exist in AWS account: company-stage
AWS user name: aadatiya does not exist in AWS account: company-dlab
AWS user name: aadatiya does not exist in AWS account: company-nonprod
AWS user name: aadatiya does not exist in AWS account: company-prod
AWS user name: aadatiya does not exist in AWS account: company-govcloud-admin-nonprod
AWS user name: abigailcharles does not exist in AWS account: company-lab
AWS user name: abigailcharles does not exist in AWS account: company-bill
AWS user name: abigailcharles does not exist in AWS account: company-stage
AWS user name: abigailcharles does not exist in AWS account: company-dlab
AWS user name: abigailcharles does not exist in AWS account: company-nonprod
AWS user name: abigailcharles does not exist in AWS account: company-prod
AWS user name: abigailcharles does not exist in AWS account: company-govcloud-admin-nonprod

I just commented out this level and it works:

  #while IFS= read -r aws_account_num
  #do
  #done

How can I do this correctly so that I loop through each name, aws account and aws account number so that each entry shows once?

bluethundr
  • 1,005
  • 17
  • 68
  • 141
  • 3
    Can you add the sample output that is expected and the sample contents of all "aws-*" files – apatniv Sep 12 '18 at 16:48
  • 3
    Well, your script is behaving correctly. For each username, you check each key. And within that, foreach key, you check every account number. Am I right in assuming that what you want to check is every username only once with the corresponding key and account number? – confetti Sep 12 '18 at 16:54
  • Thanks guys. I've updated the OP with sample output from what I want and a clearer explanation of what I'm trying to do. – bluethundr Sep 12 '18 at 17:08
  • 1
    Can you add sample *input*? – Benjamin W. Sep 12 '18 at 17:26
  • Are these files related? Are aws_user_name, aws_key, and aws_account_num correlated? In that case, you could use a single loop, not the nested ones you have now. See this post to find out how to read from three files at a time: [Looping through the content of a file in Bash](https://stackoverflow.com/a/41646525/6862601). – codeforester Sep 12 '18 at 17:33
  • it might help if you modified the first `printf` to also display `$aws_account_num` (currently not shown in your example output); if you've got multiple `aws_account_num` values to process, should that be included in your `$(aws iam get-user...)` call ... otherwise you're making the same exact call for each distinct `aws_account_num` – markp-fuso Sep 12 '18 at 17:34

3 Answers3

5

You want one loop that calls read three times, using three different file descriptors.

while IFS= read -r aws_user_name
      IFS= read -r aws_key <&3
      IFS= read -r aws_account_num <&4; do
    ...
done < "$aws_users_all" 3< "$aws_env_list" 4< "$aws_account_numbers"
chepner
  • 497,756
  • 71
  • 530
  • 681
  • Thanks! That does get me closer. However it should loop through each user name, and then all the accounts with their account numbers.What happens now is one name gets compared with an account, and then the next user name with the next account. One user account needs to get compared with ALL aws accounts before proceeding to the next AWS account. – bluethundr Sep 12 '18 at 18:06
  • This is the output that I am getting now; `AWS user name: aadatiya does not exist in AWS account: company-lab AWS user name: abigailcharles does not exist in AWS account: company-bill AWS user name: ab-jenkin does not exist in AWS account: company-stage AWS user name: ab-rrf-dev-pyservice does not exist in AWS account: company-dlab AWS user name: Administrator does not exist in AWS account: company-nonprod AWS user name: adravid does not exist in AWS account: company-prod AWS user name: aelahi does not exist in AWS account: company-govcloud-admin-nonprod` – bluethundr Sep 12 '18 at 18:07
  • You could use `&&` between each `read`, to ensure stopping the loop at *first* end-of-file. – F. Hauri - Give Up GitHub Sep 12 '18 at 18:18
  • 1
    @bluethundr Then you need a hybrid approach; an outer loop that reads from one file and an inner loop that reads from two files (or something like that). – chepner Sep 12 '18 at 18:22
  • ... and you could write `read -ru 3 aws_key` instead of `read -r aws_key <&3`. – F. Hauri - Give Up GitHub Sep 12 '18 at 18:25
  • I'm not entirely sure why `bash` bothers with that option. – chepner Sep 12 '18 at 18:29
  • I think it's for error treatment: error message is not same `read <&5 foo` -> *`bash: 5: Bad file descriptor`* and `read -u 5 foo` -> *`bash: read: 5: invalid file descriptor: Bad file descriptor`* – F. Hauri - Give Up GitHub Sep 12 '18 at 18:50
  • Simply making the redirection message more specific would work just as well: `bash: redirection from 5: Bad file descriptor`. A non-standard option isn't necessary. – chepner Sep 12 '18 at 18:56
  • @chepner Thanks! That worked. I created an outer loop and an inner loop with two variables and that is what did the trick. Please put that suggestion into the answer section so I can accept that as the answer. Thanks again! – bluethundr Sep 13 '18 at 17:29
4

There is a way using paste command:

while read aws_user_name aws_key aws_account_num ;do
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        $aws_user_name $aws_key $aws_account_num
    # ...
done < <(
    paste "$aws_users_all" "$aws_env_list" "$aws_account_numbers"
)

Sample:

User: aadatiya         Key: company-lab            AccNum: 123456789191
User: abigailcharles   Key: company-stage          AccNum: 987654321211
User: tdunphy          Key: company-nonprod        AccNum: 456721231213
User: broberts         Key: company-prod           AccNum: 123213512321

Nota: this will work simply while each file do contain 1 word by line.

... And if you plan to use space in your source files, you could use another delimiter:

while IFS=@ read -r aws_user_name aws_key aws_account_num ;do
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        "$aws_user_name" "$aws_key" "$aws_account_num"
    # ....
done < <(
    paste -d@ "$aws_users_all" "$aws_env_list" "$aws_account_numbers"
)

Or playing with IO under pure

Near @chepner solution:

exec 3<$aws_env_list
exec 4<$aws_account_numbers
while read aws_user_name ;do
    read -u 3 aws_key
    read -u 4 aws_account_num
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        "$aws_user_name" "$aws_key" "$aws_account_num"
    # ...
done <$aws_users_all
exec 3<&-
exec 4<&-

Nota: last two lines will close FD 3 and 4.

F. Hauri - Give Up GitHub
  • 64,122
  • 17
  • 116
  • 137
0

One way to do it is to load all of the input data into arrays before processing it:

IFS= mapfile -t aws_users <"$aws_users_all"
IFS= mapfile -t aws_envs <"$aws_env_list"
IFS= mapfile -t aws_acct_nums <"$aws_account_numbers"

for aws_user_name in "${aws_users[@]}" ; do
    for idx in ${!aws_envs[*]} ; do
        aws_key=${aws_envs[idx]}
        aws_account_num=${aws_acct_nums[idx]}

        # Code using $aws_user_name, $aws_key, and $aws_account_num
        ...
    done
done

Note that mapfile was introduced with Bash 4.0 so this won't work if you are using an older version of Bash. See Read lines from a file into a Bash array for alternatives that work with older versions (but note that the accepted answer is very bad).

Loading the files into memory may seem like a bad idea, but it is unlikely to be a problem in practice. Modern machines have a lot of memory, and Bash is very slow at almost everything. If there is too much data to load into memory then there is probably way too much data to process line-by-line with Bash code.

pjh
  • 6,388
  • 2
  • 16
  • 17