1

I'm trying to replicate an extremely basic manually configured AWS ECS Fargate deployment of a single container using CloudFormation. Looks like I'm almost there; the resulting stack spins up a container I can access. But there are no logs.

I compared my manual task (created via the UI) and the CloudFormation one, and added an identical log configuration to the container definition, but simply changing the log group from /ecs/foo to /ecs/bar:

LogConfiguration:
  LogDriver: awslogs
  Options:
    awslogs-create-group: true
    awslogs-group: '/ecs/bar'
    awslogs-region: !Ref AWS::Region
    awslogs-stream-prefix: 'ecs'

But now the the task fails to start a container. It gives an error like this:

Resourceinitializationerror: failed to validate logger args: create stream has been retried 1 times: failed to create Cloudwatch log group: AccessDeniedException: User: arn:aws:sts::…:assumed-role/ecsTaskExecutionRole/… is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-east-1:…:log-group:/ecs/bar:log-stream: because no identity-based policy allows the logs:CreateLogGroup action status code: 400, request id: … : exit status 1

One documentation page mentions this logs:CreateLogGroup permission, and says:

To use the awslogs-create-group option, add logs:CreateLogGroup as an inline IAM policy.

But what I don't understand is how my CloudFormation template differs from the stack manually created via the UI. By looking at the generated template for the manually-created stack, it appears both task definitions indicate the ecsTaskExecutionRole. My CloudFormation template task definition has this:

ExecutionRoleArn: 'arn:aws:iam::…:role/ecsTaskExecutionRole'

How was the manually-created stack able to create the log group, but my standalone from-scratch CloudFormation template could not? Where would I indicate the logs:CreateLogGroup permission? The manually-created stack doesn't seem to indicate any inline policy. (Admittedly for some reason the manually-created task definition doesn't seem to use a CloudFormation stack, so maybe it has some hidden settings I'm not seeing in the UI.)

Garret Wilson
  • 18,219
  • 30
  • 144
  • 272
  • Can you show what this role `arn:aws:iam::…:role/ecsTaskExecutionRole` has in it? What policies are assigned to that role? Are you sure that role has `logs:CreateLogGroup`? By default (when using the UI to create a service) I believe ECS uses the AWS provided policy: `arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy` – Mark B Feb 15 '23 at 13:09
  • I believe this role was created when I manually created the first stack (before trying to duplicate it in CloudFormation). It merely contains `AmazonECSTaskExecutionRolePolicy`, which is managed by AWS. As I mentioned I'm duplicating the JSON of the manual task definition, except in YAML in the CloudFormation. If it worked in the manual configuration, I don't see why it doesn't work in CloudFormation. – Garret Wilson Feb 15 '23 at 15:41
  • Well it's difficult to point out what could be wrong without seeing your CoudFormation YAML. – Mark B Feb 15 '23 at 15:48
  • I provided the task `LogConfiguration`. Do you have a CloudFormation template that you've tested that creates an ECS Fargate task that automatically creates the log group? If so, how did you specify the permissions? – Garret Wilson Feb 15 '23 at 18:42
  • I don't use CloudFormation, I use Terraform. I can read CloudFormation and point out what is wrong though. Just seeing the `LogConfiguration` obviously doesn't help us point out what is wrong with your IAM role. – Mark B Feb 15 '23 at 19:03
  • I'm raising this question hoping that someone who has actually done this will share how they did it, because it doesn't "just work". There are no other IAM roles in my CloudFormation template, so if you know of others that need to be added, please share them and show how to use them in CloudFormation. – Garret Wilson Feb 15 '23 at 20:30
  • I could have pointed out that you were lacking the `CreateLogGroup` permission if you had shown your CloudFormation template in your question. And I could have offered the solution of creating the log group directly in CloudFormation instead. This would have all been clear and saved you hours of time if you had provided all the code in your question instead of a tiny little piece of it. When you are encountering errors and needing help with them on here, it is expected you that show your code, instead of just waiting for someone to give you a working example. – Mark B Feb 16 '23 at 01:30
  • Thank you for the feedback, Mark, especially confirming that using `TaskRoleArn` was the wrong approach. And anyone is still free to add an answer explaining how to add the `logs:CreateLogGroup` permission inline using CloudFormation, which was part of the original question, as that is useful information to know. – Garret Wilson Feb 16 '23 at 03:41
  • I'm playing around with AWS Fargate today just to learn about it, and this same error is blocking any tasks from starting. I have exclusively used the ECS web console, and have not done anything manual with CloudFormation. Not sure if that's helpful, but thought I would contribute the data point. – duozmo Aug 17 '23 at 20:02

2 Answers2

3

If I want the task to automatically create a log group dynamically using awslogs-create-group, it appears that the correct approach is to have an IAM policy that includes the logs:CreateLogGroup permission, as mentioned at Using the awslogs log driver. (I still don't understand how creating the task definition manually in the UI resulted in the log group getting created.) Another page related to ECS resource initialization errors says I need to "add logs:CreateLogGroup as an inline IAM policy", but no one has been able to provide an example of how to do that in CloudFormation. I'm sure I could figure it out …

However rather than having the service dynamically create a log group, it seems to me better practice to declare and configure the AWS::Logs::LogGroup resource in the CloudFormation template itself. (Thanks to the AWS CloudFormation - Beginner to Advanced (Hands-On Guide) Udemy course for inspiring this approach.) Thus I would declare the log group like this in CloudFormation:

BarLogGroup:
  Type: AWS::Logs::LogGroup
  DeletionPolicy: Retain
  Properties: 
    LogGroupName: '/ecs/bar'

Then to avoid duplication, reference the log group in the log configuration:

LogConfiguration:
  LogDriver: awslogs
  Options:
    awslogs-group: !Ref BarLogGroup
    awslogs-region: !Ref AWS::Region
    awslogs-stream-prefix: 'ecs'

This gets around the problem of runtime permissions and dynamic creation altogether. The log group is a resource, too, and since the task depends on it we might as well describe it declaratively with the other resources. This way log group creation depends on permissions of the role creating the stack, not the task itself, which seems more appropriate.

(I've added a deletion policy, assuming you want to keep the logs even if you delete/recreate the stack.)

After making the above changes, my CloudFormation EC Fargate stack now runs and produces logs just like the manually created stack, so this approach is a success.

Garret Wilson
  • 18,219
  • 30
  • 144
  • 272
1

ecsTaskExecutionRole should be assigned to TaskRoleArn, not to ExecutionRoleArn.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • Oh! That sounds promising. I had to work backwards from the manually-created task, which didn't produce a stack, so I may have guessed wrong on the property. I'll try it tomorrow. – Garret Wilson Feb 15 '23 at 03:38
  • @GarretWilson No problem. Please let me know what you will find. – Marcin Feb 15 '23 at 03:39
  • Interestingly if I look at the task created via the UI, the UI label plainly says "Task execution role", but if I look down under the JSON tab, it uses `"executionRoleArn"`, which is probably why I used `ExecutionRoleArn` in the YAML. There is no `"taskRoleArn"` in the task JSON in the UI for the manually created task. – Garret Wilson Feb 15 '23 at 03:57
  • @GarretWilson The console maybe setting same role for both `ecsTaskExecutionRole` and `ExecutionRoleArn`. In fact, its a common practice anyway. – Marcin Feb 15 '23 at 04:09
  • 2
    I think this answer is incorrect. The ECS Task Execution Role is what ECS uses to interact with AWS services like ECR and CloudWatch Logs. The task role is what your own code running inside the container would use to interact with AWS. I've definitely had ECS tasks that were logging to CloudWatch Logs successfully, without any Task Role assigned to them at all. – Mark B Feb 15 '23 at 13:08
  • 2
    From more reading I agree that this answer seems incorrect. The task execution role is for actions the _task_ (the application) takes while running. Creation of a log group has to do with ECS actions, not the application. Using the same role for both ECS and the task seems to be an antipattern. See https://stackoverflow.com/a/49947471 and the fourth point at https://dashbird.io/blog/aws-ecs-containers/ . – Garret Wilson Feb 15 '23 at 14:05
  • This didn't work anyway. Changing the task role had no bearing on whether ECS was able to create the log group. (And now that I've read more this is expected—the execution role determines log group creation, not the task role.) My best guess at the moment is that somehow I need to create an inline policy in the template to allow the `logs:CreateLogGroup` permission (or create the log group manually). I'm still not sure how the `/ecs/foo` log group wound up being created when I created the task definition manually through the UI. – Garret Wilson Feb 15 '23 at 14:11