5

I have a given AWS fargate task definition. do_something calls a downstream service that does not support more than 4 concurrent consumers. I must therefore find a way to limit the number of concurrent do_something fargate tasks that run at the same time.

Is there a built-in way to do this? I wasn't able to find it in the documentation.

mabead
  • 2,171
  • 2
  • 27
  • 42

2 Answers2

2

In the scaling policy you san set the maximum number of tasks to 4.

Here you can see a picture of the setting. enter image description here

DominikHelps
  • 971
  • 4
  • 10
  • 3
    If I understand you answer correctly, this option is available when you define a "Service" from a task definition. As described here https://stackoverflow.com/a/42961623/2679757, a "Service" is something that is always running. In my case, I want to run a "Task" that runs on demand but that limits the number of concurrent execution of the task. For example, if I want to limit to 2 tasks running at the same time and I start 4 tasks, I want 2 tasks to start right now and 2 others to be queued by AWS and started when the previous tasks complete. Is it feasible with a Task? – mabead Dec 19 '18 at 18:27
  • If you use Run Task without a service you need to keep track of how many tasks you spawn yourself. For your use case I use AWS SQS as Task Queue and then spawn lambdas to work on the tasks and remove them from the queue. Lambda has the feature you are looking for:Managing Concurrency. You can limit concurrent executions of the lambdas – DominikHelps Dec 19 '18 at 18:50
  • But then your lambda must run for the same duration as your fargate task for this to work. The reason I am considering fargate is precisely to overcome the 15 minutes limitation of lambdas. I guess, I will need to use a Step functions then :( – mabead Dec 19 '18 at 18:53
  • Yeah 15 minutes is very long for a lambda. Can you split up the work in smaller pieces? Step function sounds good too – DominikHelps Dec 19 '18 at 19:03
  • My task is kind of a "for" loop where each iteration takes a few minutes. Coding a "for" loop in JSON (step function definition language) is much less intuitive than coding it in "regular" code. Thanks for all your suggestions. – mabead Dec 19 '18 at 19:14
  • 1
    Based off this answer https://stackoverflow.com/questions/66704457/limit-concurrency-of-aws-ecs-tasks it looks like _AWS Batch_ has this capability. Or for ECS Instance backed cluster you can limit your available containers, but still no answer for the OP to limit this in ECS Fargate. – Matt Cliff Feb 16 '23 at 20:22
-1

Had the same problem. Contacted support this was the answer.

Hi there, I understand that you would like to increase your Fargate limit for concurrent tasks, I'm happy to assist! As per Fargate team recommendation, can you try to launch an EC2 instance in your region for a few minutes and then terminate the instance? This process will automatically increase your concurrent task limit, if you are still experiencing issues after this, please let us know so I can contact them back in order to manually increase the limit. Have a nice day ahead!

The solution works for me..

André Alberto
  • 210
  • 1
  • 10