6

I have a task to process large amount of data, so data is batched into lots of parts. I have written the task definition for this type of work but now I know only how to set up them manually via registering multiple task definitions for each env.

sample where each task has its own env BATCH_ID

  aws ecs register-task-definition --cli-input-json file://taskdef1.json  
  aws ecs run-task --cluster $cluster --task-definition process_data_1

  aws ecs register-task-definition --cli-input-json file://taskdef2.json  
  aws ecs run-task --cluster $cluster --task-definition process_data_2

It would be even nice to have some .manifest file of all task arns put for cluster.

Is there someway to run multiple similar ECS tasks with different env params in more elegant way then creating enormous amount of different taskdefs files?

Thanks for any help and suggestions

Ivan Shelonik
  • 1,958
  • 5
  • 25
  • 49

2 Answers2

5

You can override environmental variable when you run the task with --overrides flag. I use this all the time; you can either override an existing environmental variable (defined in task definition) or simply add a new one. --overrides accepts only JSON (no shorthand syntax); in your case, it would look like:

{
  "containerOverrides": [
    {
      "name": "containerNameFromTaskDefinition",
      "environment": [
        {
          "name": "BATCH_ID",
          "value": "sampleBatchId"
        }
      ]
    }
  ]
}

and the command:

aws ecs run-task --cluster $cluster --task-definition process_data_1 --overrides {"containerOverrides":[...]}

You can use --overrides to override even more things of course: https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html

Zdenek F
  • 1,649
  • 1
  • 15
  • 27
  • Assuming you run 5 instances of the task definition how would this allow `BATCH_ID` to be 0,1,2,3,4? – Piers MacDonald Aug 13 '20 at 22:54
  • That's a good question, I thought there would be an ECS environmental variable for that, but I didn't find any ‍♂️ – Zdenek F Aug 17 '20 at 14:13
  • Is there any to option to override the env variables(Values from) ssm params with normal env variables(values) thorugh cli/json ? – Mithun Jul 01 '21 at 06:29
  • @Mithun If the override above doesn't work, I assume you can't - the documentation doesn't specify anywhere whether `secrets` take precedence over `environment` – Zdenek F Jul 01 '21 at 09:22
1

I may be misunderstanding the problem here, but could you just put the variable in your cli input .json and use sed to swap out the batch ID before you run the task? I've done this for application deployments when I need to swap out env vars.

First, put the variable you want to replace in your taskdef template file (like %BATCH_ID%)

Then do something like this: (NOTE: Bash script is off the top of my head and may need fixing)

sed 's/%BATCH_ID%/$BATCH_ID' generic_taskdef.json > process_data_$BATCH_ID.json 
aws ecs register-task-definition --cli-input-json file://process_data_$BATCH_ID.json
aws ecs run-task --cluster $cluster --task-definition process_data_$BATCH_ID
mcheshier
  • 715
  • 4
  • 13
  • Yes, I can do something like you said. But anyway it's going to create large amount of tasks, so having to pass envs o register tasks is required. I would like to know whether it's possible to use for example ecs cli compose or task definition with multiple container (for each specific envs). I want to automate this task to use only 1 button. Here is(Below) the suggestion to make different tasks for each env but it doesn't seem the simplest way. Maybe I'm wrong https://stackoverflow.com/questions/53145233/aws-ecs-start-multiple-containers-in-one-task-definition Thanks – Ivan Shelonik May 19 '19 at 18:49
  • 1
    The linked article is 'best practice' as I understand it - multiple containers in a single taskdef *should* be restricted to sidecars. If you want to make it 'push button', you're probably going to have to write a script to assemble the taskdefs (either the method I used or the method the other poster recommended with --overrides), register them, and potentially clean them up when you're done. – mcheshier May 20 '19 at 06:38