ECS Fargate Scheduled Task not running

Question

I'm trying to setup a scheduled task with ECS Fargate but I cannot figure out why it is not running. I can confirm the task works correctly using RunTask but when I try to trigger it on a schedule all I get is a bunch of 'FailedInvocations' with no explanation.

I do know though that the rule is being triggered so this is a good sign. See the screenshot below:

But everytime it is triggered there is just a 'FailedInvocation'. Here's the scheduling rule:

And the default permissions on the ecsEventRole with just ecs:runTask:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

My hunch says that this ecsEventsRole doesn't have enough permissions. Should I try to give it the ones that ecsTaskExecutionRole has?

Thanks

EDIT: This is now supported in us-east-1 region. See comments.

Scheduled Fargate tasks are not supported. Only EC2 scheduled ecs tasks — Marc Young, Aug 07 '18 at 21:43
Fargate tasks are now supported: https://aws.amazon.com/about-aws/whats-new/2018/08/aws-fargate-now-supports-time-and-event-based-task-scheduling/ — FBryant87, Sep 04 '18 at 16:30
Works in us-east-1, but unavailable in ca-central-1. Assuming there are other problem regions. — coolboyjules, Jan 18 '19 at 22:28

score 35 · Answer 1 · answered Aug 08 '19 at 07:31

35

Although it's been over an year, AWS still don't have a proper way to see invocation logs.

As you already know we can invoke tasks by RunTask manually, so does Scheduled Task.

The only difference is that Scheduled Task is triggered by CloudWatch rules.

We can easily see invoke logs in CloudTrail Event history, go there and filter events with Event name: RunTask and select the time range you want to check, find the event and click View Event, you'll see the error code and response.

answered Aug 08 '19 at 07:31

Greg

1,363
13
15

1

After I combined top 2 answer, added `ecs:RunTask` and `"iam:ListInstanceProfiles", "iam:ListRoles", "iam:PassRole"` to the cloudwatch event iam role, finally the task run successfully. – 鄭元傑 Dec 15 '20 at 03:57
This is actually the correct answer. Errors in the actual config aside, this problem was exacerbated by the logs being hidden. I had no idea Cloudtrail even existed, never mind that it ingests internal AWS events. – Stephen Wright Aug 04 '21 at 12:01

score 17 · Answer 2 · answered Jan 17 '18 at 06:49

17

I ran into a similar issue where regular ECS Scheduled Tasks were not running.

I finally resolved it by adding an additional policy to ecsEventsRole which allows CloudWatch Events to pass IAM roles to ECS Tasks:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListInstanceProfiles",
                "iam:ListRoles",
                "iam:PassRole"
            ],
            "Resource": "*"
        }
    ]
}

answered Jan 17 '18 at 06:49

Jason

301
2
4

I think this is hitting a few people because the first bit of documentation (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html) says 'You must add iam:PassRole permissions for any task role overrides to the CloudWatch IAM role', but it turns out that the task needs it to use a role already applied (explained in a different area of the documentation https://docs.aws.amazon.com/AmazonECS/latest/developerguide/CWE_IAM_role.html - 'If your scheduled tasks require the use of the task execution role or a task role override, then you must add iam:PassRole permissions'). – Chris Mar 14 '18 at 16:27
In addition to ecs:RunTask, I only needed iam:PassRole on both the Task Role and Task Execution role resources. – gileri Jan 18 '19 at 21:35

score 3 · Answer 3 · answered Jan 15 '18 at 21:14

3

Here is a possible workaround: use a lambda function as target for the cloudwatch rule and create the task in the lambda function code.

Here is an example code for the lambda function: https://lobster1234.github.io/2017/12/03/run-tasks-with-aws-fargate-and-lambda/

The links describes how to pack the new boto version with the lambda function but this is not necessary anymore since AWS already updated the lambda boto version to 1.4.8

I've tested and it works.

answered Jan 15 '18 at 21:14

Fred Liporace

387
4
12

Thats what I was trying to avoid but ended up with – Marc Young Aug 07 '18 at 21:43
I ended up using this. Probably the most reasonable solution presently. – coolboyjules Aug 08 '18 at 19:08

score 2 · Answer 4 · answered Apr 30 '20 at 13:28

When defining a Rule in CloudFormation, I had to ensure that the Target RoleArn property referenced ecsEventsRole:

Targets:
- Id: !Sub ${AWS::StackName}-cron-test-1
  Arn: arn:aws:ecs:eu-west-1:<account id>:cluster/fargate-cluster
  RoleArn: !Join
    - ':'
    - - "arn:aws:iam"
      - ""
      - !Ref "AWS::AccountId"
      - "role/ecsEventsRole"

And where ecsEventsRole defined the AmazonEC2ContainerServiceEventsRole policy

tanvi · Answer 5 · 2019-08-21T18:56:12.960

Have you tried using the aws cli and running aws events put-rule followed by aws events put-targets --rule <value> --targets <value> instead? I was having a similar problem, and using the (recent version of) the aws cli worked for me.

Here's a sample:

aws events put-rule --name "DailyLambdaFunction" --schedule-expression "cron(0 9 * * ? *)"

Followed by the below command all in one line:

aws events put-targets --rule cli-RS-rule --targets '{"Arn":"arn:aws:ecs:1234/cluster/clustername","EcsParameters":{"LaunchType": "FARGATE","NetworkConfiguration": {"awsvpcConfiguration": {"AssignPublicIp": "ENABLED", "SecurityGroups": [ "sg-id1233" ], "Subnets": [ "subnet-1234" ] }},"TaskCount": 1,"TaskDefinitionArn": "arn:aws:ecs:1234:task-definition/taskdef"},"Id": "sampleID111","RoleArn": "arn:aws:iam:1234:role/eventrole"}'

make sure to edit the security group, subnet and account ID strings before running it - they're just placeholders in the command i used above. — tanvi, Oct 01 '18 at 18:05

score 0 · Answer 6 · answered Aug 20 '20 at 11:43

I had similar problem with ECS fargate task scheduler. I've managed to fix it by adding below policy to the task IAM role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask"
            ],
            "Resource": [
                "arn:aws:ecs:*:548503722878:task-definition/test-services-20200813124716"
            ],
            "Condition": {
                "ArnLike": {
                    "ecs:cluster": "arn:aws:ecs:*:548503722878:cluster/test-cluster"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": [
                "*"
            ],
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "ecs-tasks.amazonaws.com"
                }
            }
        }
    ]
}

score 0 · Answer 7 · answered Apr 26 '23 at 20:39

I found that all the answers are missing one very important point:

The role which is used as "ecsEventsRole" (or "Target Role" if you look at it from the ECS->Scheduled_tasks) must have a Trust to the Events service in order to be able to run the task:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "events.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }
]

}

Also (as previously mentioned) it should have the permissions to run the tasks - easily achieved by adding the predefined AWS policy:

AmazonEC2ContainerServiceEventsRole

ECS Fargate Scheduled Task not running

7 Answers7