6

I am using AWS Python SDK Boto3 and I am trying to know which security groups are unused. With boto2 I did it but I do not know how to do the same with boto3.

from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo
import boto.sns
import sys
import logging
from security_groups_config import config

# Get settings from config.py
aws_access_key = config['aws_access_key']
aws_secret_key = config['aws_secret_key']    
ec2_region_name = config['ec2_region_name']
ec2_region_endpoint = config['ec2_region_endpoint']

region = RegionInfo(name=ec2_region_name, endpoint=ec2_region_endpoint)

if aws_access_key:
    conn = EC2Connection(aws_access_key, aws_secret_key, region=region)
else:
    conn = EC2Connection(region=region)

sgs = conn.get_all_security_groups()

## Searching unused SG if the instances number is 0
def search_unused_sg(event, context):
    for sg in sgs:
        print sg.name, len(sg.instances())
helloV
  • 50,176
  • 7
  • 137
  • 145
Robert
  • 10,403
  • 14
  • 67
  • 117
  • 1
    In boto3, you can gather the info from `describe_instances` and `describe_security_groups` , store both security group name value into respective set, then make a deduction. – mootmoot Dec 14 '16 at 15:53
  • Yes, of course but I was wondering if there is a function that provides this information. In boto2 there is ```get_all_security_groups()```. – Robert Dec 14 '16 at 18:37
  • Unfortunate, no. Boto3 is a rewrite API, which is well documented and well maintain, less surprise compare to boto2. – mootmoot Dec 14 '16 at 18:47
  • Boto3 has much powerful methods to get what you want. Check my answer and tweak it to your needs. – helloV Dec 14 '16 at 22:06
  • This needs an answer that finds all the use of security groups (ec2s, load balancer, lambda, more?). Actually, boto3 needs a single call way to find all references to a given security group. – Kevin Buchs Jan 28 '19 at 17:35

4 Answers4

6

Use the power of Boto3 and Python's list comprehension and sets to get what you want in 7 lines of code:

import boto3

ec2 = boto3.resource('ec2') #You have to change this line based on how you pass AWS credentials and AWS config

sgs = list(ec2.security_groups.all())
insts = list(ec2.instances.all())

all_sgs = set([sg.group_name for sg in sgs])
all_inst_sgs = set([sg['GroupName'] for inst in insts for sg in inst.security_groups])
unused_sgs = all_sgs - all_inst_sgs

Debug information

print 'Total SGs:', len(all_sgs)
print 'SGS attached to instances:', len(all_inst_sgs)
print 'Orphaned SGs:', len(unused_sgs)
print 'Unattached SG names:', unused_sgs

Output

Total SGs: 289
SGS attached to instances: 129
Orphaned SGs: 160
Unattached SG names: set(['mysg', '...
helloV
  • 50,176
  • 7
  • 137
  • 145
  • thank you for your solution but it is raising an error because ```{ "stackTrace": [ [ "/var/task/lambda_function.py", 81, "unused_sg", "sgs = list(ec2.security_groups.all())" ] ], "errorType": "AttributeError", "errorMessage": "'EC2' object has no attribute 'security_groups'" }``` – Robert Dec 15 '16 at 14:10
  • @Robert, are you sure you are using `ec2 = boto3.resource('ec2')` ? – helloV Dec 15 '16 at 14:37
  • I am creating the client: ```client = boto3.client( 'ec2', aws_access_key_id = AWS_ACCESS_KEY, aws_secret_access_key = AWS_SECRET_KEY , region_name = REGION_NAME )``` – Robert Dec 15 '16 at 17:03
  • @Robert, `client` doesn't have that method. My code uses `resource` – helloV Dec 15 '16 at 17:12
  • But before create or request the ec2 resource do I need to past the credentials configuration. ```client.resource('ec2'...)``` does not return the same object? – Robert Dec 15 '16 at 17:14
  • My code doesn't use client by you insist on using the client. Can you just copy paste the code and try? Pass the same credentials argument to resource. Or google for `boto3 resource` – helloV Dec 15 '16 at 17:17
  • How do i get all security group IDs and Related Names? together – kamal Apr 07 '18 at 20:02
4

First , I suggest you relook how boto3 deal with credential. Better use a genereic AWS credential file , so in the future when required, you can switch to IAM roles base credential or AWS STS without changing your code. 

import boto3 
# You should use the credential profile file 
ec2 = boto3.client("ec2")

# In boto3, if you have more than 1000 entries, you need to handle the pagination
# using the NextToken parameter, which is not shown here.

all_instances = ec2.describe_instances() 
all_sg = ec2.describe_security_groups()

instance_sg_set = set()
sg_set = set()

for reservation in all_instances["Reservations"] :
  for instance in reservation["Instances"]: 
    for sg in instance["SecurityGroups"]:
      instance_sg_set.add(sg["GroupName"]) 


for security_group in all_sg["SecurityGroups"] :
  sg_set.add(security_group ["GroupName"])

idle_sg = sg_set - instance_sg_set

Note : code are not tested. Please debug it as required.

mootmoot
  • 12,845
  • 5
  • 47
  • 44
  • why are people even up voting this? AttributeError: 'ec2.ServiceResource' object has no attribute 'describe_instances' – buildmaestro Aug 15 '17 at 21:30
  • 1
    @buildmaestro Check your own code. Above code is using `boto3.clent("ec2")` , NOT `boto3.resource("ec2")`. – mootmoot Aug 16 '17 at 07:58
  • @buildmaestro Please post your code as a new question to prove that. `AttributeError: 'ec2.ServiceResource' object has no attribute 'describe_instances'` is an exception thrown by boto3.resource. Check your code and see whether you accidentally redeclare the object as ServiceResource – mootmoot Aug 17 '17 at 09:22
1

Note: If you have ASG (Autoscalling group) that are in null state (count=0), when the ASG start adding adding the security groups, then it will adopt the orphan security groups. Keep in mind you need to check for the ASG security groups also

user12273105
  • 11
  • 1
0

I used an alternative approach. If we skip the credentials discussion and go back to the main question "boto3 searching unused security groups" here is an option: You go and enumerate the resource, in my case a network interface, because if you think about it, a security group has to be associated to a resource in order to be used. My example:

client = boto3.client('ec2', region_name=region, aws_access_key_id=newsession_id, aws_secret_access_key=newsession_key, aws_session_token=newsession_token)
response = client.describe_network_interfaces()
for i in response["NetworkInterfaces"]:
    #Check if the security group is attached
    if 'Attachment' in i and i['Attachment']['Status'] == 'attached':
        #Create a list with the attached SGs
        groups = [g['GroupId'] for g in i['Groups']]

II used the network interface resource because I needed to get public IPs for the accounts.

Belial
  • 821
  • 1
  • 9
  • 12