-1

I have deployed a selenium script on ECS Fargate which communicates with my server through API. Normally almost 300 scripts run at parallel and bombard my server with api requests. I am facing Net::Read::Timeout error because server is unable to respond in a given time frame. How can I limit ecs tasks running at parallel. For example if I have ran 300 scripts, 50 scripts should run at parallel and remaining 250 scripts should be in pending state.

Armaghan Fazal
  • 43
  • 2
  • 9

2 Answers2

1

I think for your use case, you should have a look at AWS Batch, which supports Docker jobs, and job queues.

This question was about limiting concurrency on AWS Batch: AWS batch - how to limit number of concurrent jobs

Edit: btw, the same strategy could maybe be applied to ECS, as in assigning your scripts to only a few instances, so that more can't be provisioned until the previous ones have finished.

Romain Prévost
  • 513
  • 2
  • 12
  • Thanks Romain, yeah I have used a similar technique. `sleep(3) while client.list_tasks(cluster: ClusterName).task_arns.count >= MAX_CONCURRENT_COUNT` – Armaghan Fazal Mar 25 '21 at 14:34
0

I am unclear how your script works and there may be many ways to peal this onion but one way that would be easier to implement assuming your tasks/scripts are long running is to create an ECS service and modify the number of tasks in it. You can start with a service that has 50 tasks and then update the service to 20 or 300 or any number you want. The service will deploy/remove tasks depending on the task count parameter you configured.

This of course assumes the tasks (and the script) run infinitely. If your script is such that it starts and it ends at some point (in a batch sort of way) then probably launching them with either AWS Batch or Step Functions would be a better approach.

mreferre
  • 5,464
  • 3
  • 22
  • 29
  • 1
    Thank you for your response. My script runs for few minutes and then terminates. Service is for infinite running tasks which can not be used in this scenario. Is there any way in aws batch to limit its concurrent running jobs? – Armaghan Fazal Mar 19 '21 at 13:10
  • Ok yeah then ECS Services are out of discussion. I don't know Batch inside-out but @romain in his answer provided a link to another thread where they say it's possible. – mreferre Mar 19 '21 at 14:40
  • `sleep(3) while client.list_tasks(cluster: ClusterName).task_arns.count >= MAX_CONCURRENT_COUNT` **client** is an object of **Aws::Ecs**. I have used this jugaad for now. P.S I am looking how AWS Batch can help me in this use case – Armaghan Fazal Mar 24 '21 at 05:58