AWS CloudWatch Rule returns FailedInvocation with AWS batch as Target

Question

Hi I've scheduled a Cloudwatch rule in order to run every Wednesday at 14.15 GTM by having as target an AWS Batch, which always returns FailedInvocation. I'm seeing the FailedInvocation event from associated metrics

However there are no logs regarding the error, I cannot understand the problem.

I've followed this tutorial: https://docs.aws.amazon.com/batch/latest/userguide/batch-cwe-target.html I'm stucked here from hours any suggestion?

CONFIGURATIONS

The AWS batch target is configured as is:

job Queue= arn:..
job Definition = arn:...
Job Name = name

The role associated to the target has the following policies:

arn:aws:iam::aws:policy/service-role/AWSBatchServiceEventTargetRole

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "batch:SubmitJob"
        ],
        "Resource": "*"
    }
  ]
  }

arn:aws:iam::216314997889:role/awsInvokeActionOnEc2

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "cloudwatch:Describe*",
            "ec2:Describe*",
            "ec2:RebootInstances",
            "ec2:StopInstances",
            "ec2:TerminateInstances"
        ],
        "Resource": "*"
    }
]
}

and Trust relationships

{
"Version": "2012-10-17",
"Statement": [
  {
    "Effect": "Allow",
    "Principal": {
      "Service": "events.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
  }
 ]
}

HI, yes by using CloudTrail I've found that I was using a wrong ARN job definition. — Sashimi, Jun 20 '18 at 09:04

score 6 · Accepted Answer · answered May 09 '18 at 10:33

6

Enable CloudTrail to find out the FailedInvocation reason in its logs. I agree going through CloudTrail to find out the failure reason is terrible. But for now, that's all there is. Faced the same issue and found out the Input

answered May 09 '18 at 10:33

erezool

76
1

12

I've found your suggestion very helpful, btw for the ones who read to find informations about submitting job status via cloudwatch rule: Go to `CloudTrail -> Event History` and filter by `Event Source: batch.amazonaws.con` you will see appearing rows with `EventName=SubmitJob`, click on *View Event*. it will open a json object, scroll down until you found **"responseElements** and **errorCode** to view the problem – Sashimi Jun 22 '18 at 11:29
You saved my day guys thanks :) – Nicoowr Oct 26 '21 at 14:33

score 4 · Answer 2 · answered Jan 08 '22 at 07:05

If anyone ever encounters FailedInvocations from event rules targeting Cloudwatch log groups, this is most likely due to the absence of a "Cloudwatch log resource policy" permitting the AWS Events service to create Cloudwatch logs. If you create the rule through the console, there should be an appropriate one automatically provisioned. You can check whether you have one provisioned:

aws logs describe-resource-policies

If you already have an appropriate Cloudwatch log resource policy configured, you should see something like:

{
    "resourcePolicies": [
        {
            "policyName": "TrustEventsToStoreLogEvents",
            "policyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"TrustEventsToStoreLogEvent\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":[\"events.amazonaws.com\",\"delivery.logs.amazonaws.com\"]},\"Action\":[\"logs:PutLogEvents\",\"logs:CreateLogStream\"],\"Resource\":\"arn:aws:logs:eu-central-1:1234567890:log-group:/aws/events/*:*\"}]}",
            "lastUpdatedTime": 1641611871623
        }
    ]
}

However, if you've configured your rules with Terraform (maybe even Cloudformation), then this will probably not automatically be provisioned.

Here's an example Terraform excerpt to provision a policy matching the one auto-configured through the console:

data "aws_iam_policy_document" "events_delivery_logs_write_logs" {
  statement {
    sid = "TrustEventsToStoreLogEvent"

    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = ["arn:${data.aws_partition.current.partition}:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/events/*:*"]

    principals {
      identifiers = [
        "events.amazonaws.com",
        "delivery.logs.amazonaws.com"
      ]
      type = "Service"
    }
  }
}

resource "aws_cloudwatch_log_resource_policy" "events_delivery_logs_write_logs" {
  policy_document = data.aws_iam_policy_document.events_delivery_logs_write_logs.json
  # This is the standard name this is utilized when creating a CW event rule -> CW log group through the console
  policy_name = "TrustEventsToStoreLogEvents"
}

Infra resources:

score 0 · Answer 3 · answered Sep 11 '20 at 21:29

If you are looking for the reason your invocations are failing, see the other answers UNLESS you're trying to implement AWS::Events::Rule and you're seeing failed invocations. The following answer may resolve the issue and negate to need to find these non-existent logs.

Cloudwatch failedinvocation error no logs available

AWS CloudWatch Rule returns FailedInvocation with AWS batch as Target

CONFIGURATIONS

3 Answers3