4

I have set up an ECS cluster backed up by AWS Fargate. Assigned minimal CPU (256) and Memory (512) in order to test scale in and scale of for this cluster. This cluster scales and scales out successfully depending on the load. However, the scale out operation triggers approximately 10 minutes after the load reaches more than 50% (which is AutoScalingTargetValue for CPU usage). Similarly, it takes approximately 20 minutes in order to scale in after load drops to 15% CPU usage.

I am not understanding how should I configure the Autoscaling group in order for the policy to respond within a minute to changing load.

kk.
  • 3,747
  • 12
  • 36
  • 67

1 Answers1

3

It sounds like you need to configure your autoscaling rules to be a bit faster. Note that autoscaling rules are triggered by CloudWatch alarms, and they will only take effect when the alarm goes off. Check to make sure that the alarm doesn't have a long alarm period or cooldown period, as this would cause the alarm to wait a while before going off, or go on cooldown for a while before adjusting again.

nathanpeck
  • 4,608
  • 1
  • 20
  • 18
  • Thanks @nathanpeck for your valuable input. I did look at this cloudwatch metric for High and Low triggers. By default they were configured to 3 events in 3 minutes span for scale up and 15 events in 15 minutes for scale down. I have reduced it to 1 event in 1 minute for both scale up and scale down. This configuration has improved the timings. However, it still takes 6 minutes consistently to tear down each container even if the load is 0. – kk. Apr 16 '19 at 19:53
  • 1
    Check the "deregistration delay" setting on your load balancer target group. (https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay) By default it waits up to 5 minutes for connections to close in case there are any in-flight requests. You can reduce this draining phase to a much lower value in most cases. I generally set it to only 30 seconds. This will greatly speed up teardown – nathanpeck Apr 16 '19 at 21:52
  • 1
    Thanks @nathanpack. This works for me. I have manually updated cloudwatch Alarm High/Low values to 1 datapoint in 1 minute. What kind of configuration is needed in CloudFormation template to set these values automatically? Do you want me to raise another question for it? – kk. Apr 17 '19 at 10:32
  • Yeah that would probably best be its own question – nathanpeck Apr 17 '19 at 16:55
  • @kk. : did you open another SO question for the CloudFormation setup? If so, can you link it here? I’m running into the exact same thing: on CF, when I create my own alarms, they trigger but the scaling policy actions fail. – Pierre Apr 30 '21 at 12:27
  • @Pierre Nope, I don't think I've raised a query. If you post your question, I would be able to post the code snippet for your reference. – kk. Apr 30 '21 at 16:31
  • @kk.: I asked the question on how to set that up in CloudFormation in this new question: https://stackoverflow.com/questions/67335419/cloudformation-ecs-fargate-autoscaling-target-tracking-1-custom-alarm-in-1-minu – Pierre Apr 30 '21 at 16:37