0

I am trying to diagnose an issue where I am hitting service limits on ECS. My question is, how do I diagnose a service limit issue to figure out exactly which service limit I need to increase?

The error that I ran into doesn't really provide any details on which limit I hit, just that it was ECS:

Rate exceeded (Service: AmazonECS; Status Code: 400; Error Code: ThrottlingException; Request ID: 48f19207-b486-44fb-8cec-3d5e694392e6; Proxy: null)

Another thing to note, is that this is ECS on Fargate so wondering if I am possibly exceeding a Fargate quota.

I've reviewed the ECS service quotas for my account and it doesn't seem like I should be hitting one.

Is there a way I can find out exactly which limit I hit?

z3plike
  • 15
  • 4
  • That's not a service quota error, that's an API call rate limit issue. What **specifically** are you doing when you get that message? Also this: "Another thing to note, is that this is ECS on Fargate so wondering if I am possibly exceeding a Lambda quota." makes no sense. ECS and Fargate are unrelated to Lambda. – Mark B Jul 14 '23 at 23:16
  • Also, where exactly does the error message appear? – gshpychka Jul 15 '23 at 08:23
  • Fixed typo stating lambda, meant Fargate. @mark, are you sure it's an API call rate limit? If so how did you discover that? – z3plike Jul 17 '23 at 21:36
  • @gshpychka I am getting this error when launching hundreds of ECS tasks at once, I think it is related to the run-task API. https://stackoverflow.com/questions/53161737/fargate-throttlingexception-rate-exceeded?rq=2 – z3plike Jul 17 '23 at 21:37
  • 1
    It says "Rate exceeded" not "Quota exceeded". The rate is the amount of times you are calling ECS to create tasks within a given time period. Here is the documentation you need to read https://docs.aws.amazon.com/AmazonECS/latest/APIReference/request-throttling.html – Mark B Jul 17 '23 at 22:52
  • @MarkB that should probably be an answer – gshpychka Jul 18 '23 at 10:04

1 Answers1

0

That's not a service quota error, that's an API call rate limit issue. It says "Rate exceeded" not "Quota exceeded". The rate is the amount of times you are calling ECS to create tasks within a given time period. Please see the documentation on the ECS API rate limits.

I am getting this error when launching hundreds of ECS tasks at once

Per the documentation I linked, the RunTask API (when launching EC2 backed tasks) has a maximum burst rate of 100 requests per second, and a sustained rate of 40 per second. When launching to AWS Fargate, there is a hard limit of 20 requests per second.

You can request an increase for API throttling quotas for your AWS account via the AWS Support Center.

Amazon recommends implementing an exponential backoff algorithm for API calls that may encounter throttling. Amazon has posted some articles and examples of implementing this algorithm. Those are linked at the bottom of the documentation I linked.

Mark B
  • 183,023
  • 24
  • 297
  • 295