Using the CIDR of the VPC in the inbound section of instance security group stops load balancer reaching it

Question

I have a network load balancer and an instance both in the same subnet. I want to restrict the inbound traffic of the instance to only come from the VPC (so internet cant reach it). So I used the VPC's CIDR to do this. However, this doesn't work. But if I change this CIDR group in the security group to allow all addresses ('0.0.0.0/0') then it does work. But they are in the same VPC and subnet so it shouldn't make a different.

The instance is a proxy and I'm testing it using python requests, and adding the load balancer's address as the proxy like this:

requests.get('https://www.google.com' , proxies={'https':'<load_balancer_address>'})

It must be using the load balancer and not the instance directly because changing the address at all in the line above makes it block, meaning it definitely using that address as proxy for the request.

My terraform code which describes my whole setup is below.

The part I'm referring to is the ingress for port 3128 in aws_security_group.instance_sg. Changing this CIDR group to 0.0.0.0/0 makes it work. But I don't understand why it's not working already.

The health check on port 54321 (which also comes from the load balancer) is working correctly and that has the same CIDR group.

provider "aws" {
    region = "eu-west-2"
}

resource "aws_vpc" "main" {
  cidr_block       = "172.31.0.0/16"
  instance_tenancy = "default"

  tags = {
    Name = "main"
  }
}

resource "aws_network_acl" "main" {
  vpc_id = aws_vpc.main.id

  ingress = [
    {
      from_port  = 22
      to_port    = 22
      protocol   = "tcp"
      rule_no    = 100
      action     = "allow"
      cidr_block = "${chomp(data.http.myip.body)}/32"
      ipv6_cidr_block = ""
      icmp_type = 0
      icmp_code = 0
    },
    {
      from_port  = 3128
      to_port    = 3128
      protocol   = "tcp"
      rule_no    = 200
      action     = "allow"
      cidr_block = "${chomp(data.http.myip.body)}/32"
      ipv6_cidr_block = ""
      icmp_type = 0
      icmp_code = 0
    }
  ]

  egress = [
    {
      from_port  = 443
      to_port    = 443
      protocol   = "tcp"
      rule_no    = 100
      action     = "allow"
      cidr_block = "0.0.0.0/0"
      ipv6_cidr_block = ""
      icmp_type = 0
      icmp_code = 0
    }
  ]

  tags = {
    Name = "main"
  }
}

resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "main"
  }
}

resource "aws_route_table" "public_routes" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }

  tags = {
    Name = "public_routes"
  }
}

resource "aws_subnet" "public_zone" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "172.31.0.0/20"
  availability_zone = "eu-west-2a"
}

resource "aws_route_table_association" "public_zone_assoc" {
    subnet_id = aws_subnet.public_zone.id
    route_table_id = aws_route_table.public_routes.id
}

data "http" "myip" {
  url = "http://ipv4.icanhazip.com"
}

resource "aws_security_group" "bastion" {
  vpc_id = aws_vpc.main.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["${chomp(data.http.myip.body)}/32"]
  }

  egress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [ "172.31.0.0/16"]
  }
}

resource "aws_instance" "bastion" {
    ami = "ami-0194c3e07668a7e36"
    instance_type = "t2.micro"
    security_groups = [aws_security_group.bastion.id]
    tags = {
      "Name" = "Bastion"
    }
    subnet_id = aws_subnet.public_zone.id
    associate_public_ip_address = true
    key_name = "pumpbot"
}

resource "aws_security_group" "instance_sg" {
  vpc_id = aws_vpc.main.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    security_groups = [aws_security_group.bastion.id]
  }

  ingress {
    from_port   = 3128
    to_port     = 3128
    protocol    = "tcp"
    cidr_blocks = ["172.31.0.0/16"]
  }

  ingress {
    from_port   = 54321
    to_port     = 54321
    protocol    = "tcp"
    cidr_blocks = ["172.31.0.0/16"]
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "proxies" {
    count = 1
    ami = "ami-0ef0d632eb136502d"
    instance_type = "t2.micro"
    security_groups = [aws_security_group.instance_sg.id]
    tags = {
      "Name" = "terraformproxy-${count.index + 1}"
    }
    subnet_id = aws_subnet.public_zone.id
    key_name = "pumpbot"
    associate_public_ip_address = true
}

resource "aws_lb_target_group" "tg" {
  name     = "lb-target-group"
  port     = 3128
  protocol = "TCP"
  vpc_id   = aws_vpc.main.id

  health_check {
    port = 54321
  }
}

resource "aws_lb_target_group_attachment" "tga" {
  count = length(aws_instance.proxies)
  target_group_arn = aws_lb_target_group.tg.arn
  target_id        = aws_instance.proxies[count.index].id
  port             = 3128
}

resource "aws_lb" "pump_bot_lb" {
  name               = "pump-bot-lb"
  load_balancer_type = "network"
  subnets = [aws_subnet.public_zone.id]

  enable_cross_zone_load_balancing   = true

  tags = {
    Name = "pump-bot-lb"
  }
}

resource "aws_lb_listener" "lb_listener" {
  load_balancer_arn = aws_lb.pump_bot_lb.arn
  port              = "3128"
  protocol          = "TCP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tg.arn
  }
}

Answer 1

Network Load Balancers pass through the traffic as-is. The instance will see the traffic as if it came from the client directly. The traffic won't have the Network Load Balancer's IP address associated with it. That's why you have to set the security group rule to 0.0.0.0/0 for your EC2 instance to accept the incoming traffic.

For further details you might read the answers to this question.