1

I'm having trouble while trying to apply a modification on a ECS cluster. Particularities of the environment:

  • Cluster has 2 services: blue and green.
  • Currently both services are associated to an Application Load Balancer that is exposed to the Internet.

What I want to do: add another ALB, in this case an internal one, to receive requests from private subnets in the VPC for the same service (same container, same port). When trying to apply those modifications I'm getting the following error:

CloudFormation cannot update a stack when a custom-named resource requires replacing. Rename [ClusterName|ServiceName] and update the stack again.

I'm describing these new entities using yml files. It's worth to mention that the new load balancer, its listeners and target groups were created successfully (even though the target group is not detecting the EC2 instances). The problem occurs when adding the LB to the the ECS service. Is this normal? Is it possible to have 2 LBs for the same ECS service for same port and same container name? Is there a workaround for doing this without renaming the cluster?

Edit: I tried creating a new ECS service with 2 load balancers associated to it and I got the following error (which is much more specific):

load balancers can have at most 1 items

So no, ECS services can't be associated to more than one ALB. The remaining question is: is there a workaround for this other than creating new ECS services for private subnet use?

Thanks.

ggimenez
  • 65
  • 1
  • 9

4 Answers4

4

On the 30th of July 2019, Amazon ECS released support for working with multiple load balancers / target groups in an ECS service. From their What's New blog post:

Amazon ECS services now support multiple load balancer target groups

You can now attach multiple target groups to your Amazon ECS services that are running on either Amazon EC2 or AWS Fargate. Target groups are used to route requests to one or more registered targets when using a load balancer. Attaching multiple target groups to your service allows you to simplify infrastructure code, reduce costs and increase manageability of your ECS services.

As described in the docs, this enables different setups, including integration with internal and external load balancers as you mention in your question. From the docs (emphasis mine):

Example: Having separate load balancers for internal and external traffic.

In the following use case, a service uses two separate load balancers, one for internal traffic and a second for internet-facing traffic, for the same container and port.

"loadBalancers":[
   //Internal ELB
   {  
      "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456",
      "containerName":"nginx",
      "containerPort":8080
   },
   //Internet-facing ELB
   {  
      "targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321",
      "containerName":"nginx",
      "containerPort":8080
   }
]

Related (and now closed) github issues from the public roadmap of the AWS container services are found here:

Lasse Christiansen
  • 10,205
  • 7
  • 50
  • 79
1

As you observed correctly, ECS has a limit of 1 load balancer per service. [1]
Another SO thread states the following: [2]

It is not possible to for an Elastic Load Balancer to have both a public IP address and a private IP address. It is one or the other, but not both.

If you want your ELB to have a private IP address, then it cannot listen to requests from the internet.

That means, you could use a single public load balancer for instances in your private subnet, if they are NATed and thus have access to the internet. However, as that is probably not the case, you could use two load balancers - each backed by a respective ECS service. I know that is not what you want...

So, looking for a workaround, I discovered the following solution which might work under certain circumstances (see 'prerequisites'):

Prerequisites

  1. You do not need to load balance internal traffic from you public subnet, but can access the tasks directly instead (in fact load balancing is provided by Route53).
  2. You are using the FARGATE launch type.
  3. You use the awsvpc, bridge, or host network mode.
  4. You do not use a Classic Load Balancer.

Solution

Use service discovery for ECS. [3]
It is an integration of AWS Cloud Map into AWS ECS.

You can attach your public load balancer to the ECS service as before. Additionally, you set up a service discovery namespace for the service.
ECS writes the tasks' private IPs into a DNS namespace. The DNS namespace can then be queried by the instances in your private subnet.

I think this solution should work because the docs explicitly state:

You can configure service discovery for an ECS service that is behind a load balancer, but service discovery traffic is always routed to the task and not the load balancer. [3]

Please note the following if you want to set this scenario up:

Service discovery can only be configured when first creating a service. Updating existing services to configure service discovery for the first time or change the current configuration is not supported. [3]

References

[1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_limits.html ("Number of load balancers per service")
[2] https://stackoverflow.com/a/36586238/10473469
[3] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

Community
  • 1
  • 1
Martin Löper
  • 6,471
  • 1
  • 16
  • 40
  • Thank you for your detailed answer, Martin. Finally we just duplicated the service, that is what AWS support guys recommended us for this case and, even though it's not ideal, it's not so dirty either – ggimenez Jul 27 '19 at 04:20
  • I just saw that AWS added new features to ECS and its ELB integration as Lasse pointed out correctly in the accepted answer. This makes my answer completly outdated. Just adding this note here to avoid confusion among readers – Martin Löper Apr 04 '20 at 15:24
1

I am late though but the feature launched in july can solve this problem for the people who are looking for it.

Ref: https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecs-services-now-support-multiple-load-balancer-target-groups/

Now, you can attach multiple target groups per ECS service. This allows you to maintain a single ECS service that can serve traffic from both internal and external load balancers and support multiple path-based routing rules and applications that need to expose more than one port.

mohit
  • 2,325
  • 23
  • 48
0

We have finally decided to have 2 services: one for internal use and one for public use, each one of those with its own load balancer (internal and internet-facing, respectively). Support guys from AWS approved this solution afterwards, they said this is the recommended workaround for this case.

ggimenez
  • 65
  • 1
  • 9