sort_by function when using AWS CLI for aws ec2 describe-security-groups

Question

I am trying to compare 2 security groups in AWS that are in 2 different accounts, using aws ec2 describe-security-groups. Even if the security groups are exactly the same, it's impossible for me to detect that because the json response is sorted differently in the 2 accounts.

SG_1=$(aws ec2 describe-security-groups \
--filters Name="tag:Name",Values="MainSG" \
--query 'SecurityGroups[].{
IpPermissions:IpPermissions[].{
                IpProtocol: IpProtocol,
                IpRanges: IpRanges[].{CidrIp:CidrIp},
                FromPort: FromPort,
                ToPort: ToPort
},
IpPermissionsEgress:IpPermissionsEgress[].{
                IpProtocol: IpProtocol,
                IpRanges: IpRanges[].{CidrIp:CidrIp},
                FromPort: FromPort,
                ToPort: ToPort
}
}' \
--output json)

SG_2=$(aws ec2 describe-security-groups \
--filters Name="tag:Name",Values="MainSG" \
--query 'SecurityGroups[].{
IpPermissions:IpPermissions[].{
                IpProtocol: IpProtocol,
                IpRanges: IpRanges[].{CidrIp:CidrIp},
                FromPort: FromPort,
                ToPort: ToPort
},
IpPermissionsEgress:IpPermissionsEgress[].{
                IpProtocol: IpProtocol,
                IpRanges: IpRanges[].{CidrIp:CidrIp},
                FromPort: FromPort,
                ToPort: ToPort
}
}' \
--output json)

if "$SG_1" = "$SG_2" ]; then
    echo "SAME"
else
    echo "DIFFERENT"
fi

The output for SG_1 is :

[
  {
    "IpPermissions": [
      {
        "IpProtocol": "-1",
        "IpRanges": [
          {
            "CidrIp": "10.192.55.56/32"
          },
          {
            "CidrIp": "10.111.11.0/26"
          }
        ],
        "FromPort": null,
        "ToPort": null
      },

The output for SG_2 is :

[
  {
    "IpPermissions": [
      {
        "IpProtocol": "-1",
        "IpRanges": [
          {
            "CidrIp": "10.111.11.0/26"
          },
          {
            "CidrIp": "10.192.55.56/32"
          }
        ],
        "FromPort": null,
        "ToPort": null
      },

I have tried to use jq but the response is still sorted differently. Here is the jq command I have tried to use :

jq '.[0].IpPermissions |= sort_by(.IpRanges[].CidrIp) | .[0].IpPermissionsEgress |= sort_by(.IpRanges[].CidrIp)')

I have also tried something simple :

aws ec2 describe-security-groups --query 'sort_by(SecurityGroups, &CidrIp)[].{CidrIp: CidrIp}'

In function sort_by(), invalid type for value: expected one of: ['string', 'number'], received: "null"

Thanks, indeed I forget the "[" when I posted but unfortunately this is not my main issue — Zak25, Aug 28 '23 at 16:34
What is different about the two outputs? Can you share details, without exposing anything confidential. — jarmod, Aug 28 '23 at 16:47
Related: [How to sort a json file by keys and values of those keys in jq](https://stackoverflow.com/questions/30331504/how-to-sort-a-json-file-by-keys-and-values-of-those-keys-in-jq). — jarmod, Aug 28 '23 at 17:23

score 0 · Accepted Answer · answered Aug 28 '23 at 19:03

If you're just trying to sort items in the IpRanges fields, you could do this:

.[].IpPermissions[].IpRanges |= sort_by(.CidrIp)

You should be able to then compare for deep equality. You'll probably want to do this for all arrays that may be ordered differently.

score 0 · Answer 2 · answered Aug 28 '23 at 19:12

Try this :

#!/bin/bash

security-groups(){
aws ec2 describe-security-groups \
    --filters Name="tag:Name",Values="$1" \
    --query 'SecurityGroups[].{
    IpPermissions:IpPermissions[].{
                    IpProtocol: IpProtocol,
                    IpRanges: IpRanges[].{CidrIp:CidrIp},
                    FromPort: FromPort,
                    ToPort: ToPort
    },
    IpPermissionsEgress:IpPermissionsEgress[].{
                    IpProtocol: IpProtocol,
                    IpRanges: IpRanges[].{CidrIp:CidrIp},
                    FromPort: FromPort,
                    ToPort: ToPort
    }
    }' \
    --output json \
| jq '.[][][].IpRanges |= sort_by(.CidrIp)'
}

sg=MainSG

if diff -q <(security-groups $sg) <(security-groups $sg); then
    echo "SAME"
else
    echo "DIFFERENT"
fi

You need to adapt shell function security-groups in order to get raw data from two different AWS accounts.

sort_by function when using AWS CLI for aws ec2 describe-security-groups

2 Answers2