Can't get AWS Lambda function to log (text output) to CloudWatch

Question

I'm trying to set up a Lambda function that will process a file when it's uploaded to an S3 bucket. I need a way to see the output of console.log when I upload a file, but I can't figure out how to link my Lambda function to CloudWatch.

I figured about by looking at the context object that my log group is /aws/lambda/wavToMp3 and the log stream is 2016/05/23/[$LATEST]hex_code_redacted. So I created that group and stream in CloudWatch, yet nothing is being logged to it.

Can you log the context.logGroupName, and context.logStreamName to lamda console? and then confirm whether you are checking the correct stream. — Shibashis, May 23 '16 at 05:03
Yes, I did that.. that's how I got the group & stream in the first place. — ffxsam, May 23 '16 at 05:05
Interesting typically this does not need any additional configuration, have you tried navigating to the cloudwatch console from aws lambda monitoring screen, there is a link on the top rightside? — Shibashis, May 23 '16 at 05:08
Yes, tried that as well. It takes me to CloudWatch but there is no event data. — ffxsam, May 23 '16 at 05:10
Only reason i can think of is lack of permission for the lambda execution role? can you confirm the role has following permissions "logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents". — Shibashis, May 23 '16 at 05:14
@Shibashis YES. That was it. Make that your official answer and I'll accept it. — ffxsam, May 23 '16 at 05:33
Similar to `arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole` — Ben, Dec 01 '20 at 22:16

score 204 · Answer 1 · edited Sep 11 '19 at 06:47

204

After you update your policy, it seems that you have to update your function's settings to refresh all job instances to read new policies.

So if you just click 'test' button from Lambda console after you update your role policy in IAM, the cached Lambda instances will still have old role permissions, so you will still see no logs being written to Cloudwatch logs.

Just change your timeout by a second and click on 'save and test' button, and you will start to see logs in Cloudwatch.

edited Sep 11 '19 at 06:47

Anand Tripathi

14,556
1
47
52

answered Jan 26 '17 at 12:47

hoonoh

2,151
1
11
10

21

Thanks! I was stuck on this for an hour. – reggie3 May 29 '17 at 01:18
8

This is my vote for the correct answer. Nothing in the documentation will help if you're running into this AWS bug. – brycem Jun 11 '17 at 02:52
2

Thanks for this... this is where I was stuck as well. – John Chrysostom Feb 27 '18 at 14:34
1

Thanks ! They should at least add a 'reload role/policy' in the lambda role editor so you don't need to do a dummy edit. Or even an option to 'propagate' a policy update. – Martin Sep 20 '18 at 16:19
2

It'd be great if this answer could be merged into the accepted one - they're part of the same process necessary to solve the issue. – Michael Berry Oct 03 '18 at 09:49
1

This solved it for me as well!! Had all the right IAM policies but without this, no logs were showing. Thanks a lot for this tip – bbop99 Nov 03 '18 at 12:25
Awesome, thank you. It worked for me. My guess is that there is something related to the container in which the Lambda is running - because of that, the function probably does not reload the role and its policy(s). – George Smith Nov 05 '19 at 07:30
This answer made me try the "Test" button and then the log appeared, thank you. – Alexander Santos Aug 13 '21 at 21:03
definitely useful info to know, albeit, as a noob to aws I feel it's not complete. What do you mean by "change your timeout by a second"? That's not clear, and there are like a billion settings in aws, so I don't know where or how I'm to find what you're talking about @hoonoh – DeltaPng Apr 25 '23 at 09:35
@DeltaPng This answer is quite old, and the AWS console UI for Lambda has change quite a bit, and as far as I remember the timeout config used to be very easy to find. By changing the timeout by as second, it would have least changes affecting runtime setting for the Lambda function and IAM policy would be re-loaded. – hoonoh Apr 25 '23 at 11:49

score 170 · Accepted Answer · edited Aug 10 '23 at 01:56

170

For the Lambda function to be able to create a log group and publish logs to that group, the execution role needs to have the following permissions:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": "logs:CreateLogGroup",
        "Resource": "arn:aws:logs:region:accountID:*"
    },

    {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": [
            "arn:aws:logs:region:accountID:log-group:/aws/lambda/functionname:*"
        ]
    }
]
}

Reference: https://docs.aws.amazon.com/lambda/latest/operatorguide/access-logs.html

edited Aug 10 '23 at 01:56

N R

333
3
14

answered May 23 '16 at 05:38

Shibashis

8,023
3
27
38

18

Note the answer below by hoonoh, I needed to update settings before this new policy actually applied. – Steve Smith Jun 23 '17 at 18:42
6

I came here because I faced the same problem, in my case the error was `Log group not found`. Adding it here so anyone else googling for it will find this answer – hansaplast Feb 04 '18 at 09:29
1

As said by @SteveSmith, look at hoonoh answer below (which has more votes than this accepted answer) : it only works if you update the lambda after you update its policy (modifying the timeout, adding a dummy environment var, etc...) – Christophe Blin Nov 16 '18 at 09:54

score 21 · Answer 3 · answered Jan 31 '17 at 03:26

21

For the lambda function to create log stream and publish logs to cloudwatch, the lambda execution role needs to have the following permissions

I already had these permissions yet it did not work.

Just change your timeout by a second and click on 'save and test' button, and you will start to see logs in Cloudwatch.

I changed the timeout, saved and logs still did not work.

I assigned another role and logs still did not work.

What ended up working for me was clicking "Create a custom role", then "Allow". This was it and logs started being generated but since I did not want to use a new role but my existing role, I simply assigned my existing role afterwards and it worked. So technically I should have returned back to original configuration that did not work but now it works. Go figure.

answered Jan 31 '17 at 03:26

tomusiaka

4,142
1
14
5

4

Same here, spent like 2 hours trying to solve this. Disappointed with how buggy and absolutely not transparent and not intuitive the lambda-to-cloudwatch setup is. Thanks! – alecxe Jun 22 '17 at 15:10
1

This is awful. Someone should seriously be competing with AWS Lambda. It's like they have no incentive to improve the service. – Cameron A. Ellis Apr 12 '18 at 03:41
1

Happened to me, too, and I'm pretty sure it was because I changed the execution role to an existing role which did not have explicit permissions to create/write to the Cloudwatch log stream for my lambda. – user2719094 Mar 26 '19 at 06:43
Thanks! It Worked! – Krishna Oct 24 '19 at 16:43
what do you mean by "allow"? Just a generic role? – coderboi Mar 29 '21 at 18:41

score 9 · Answer 4 · answered Jul 22 '20 at 21:59

9

July 2020 Update !!

Logs may not be in us-east-1, try looking for lambda edge logs in different regions !!

answered Jul 22 '20 at 21:59

jellycsc

10,904
2
15
32

Thanks! In my case when I send a request with Test button on AWS UI logs are created in us-east-1, but when I send a request via my application logs are created in some other region. – afsinka Jul 05 '21 at 13:49
2

Thanks - this was the case for me also. Despite all the AWS docs saying the logs will end up in the same region as your Lambda function executes, mine were in London - local to me - with the path prefix `/aws/lambda/us-east-1.`... go figure! – mustdobetter Aug 13 '22 at 14:47

score 8 · Answer 5 · answered Feb 09 '18 at 00:47

8

Apparently another necessity for logging to happen is the Lambda function must indicate completion; for instance in the Python context, the handler must return something other than None.

answered Feb 09 '18 at 00:47

Vic

593
7
11

score 4 · Answer 6 · answered Sep 13 '18 at 12:38

As other answers state you need to give lambda permission to post logs to cloud watch logs. AWS had provided AWSLambdaExecute policy just for that. It's json is -

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

You can add this policy in your role which is assigned to your lambda and you should start seeing the logs.

NOTE: It also has S3 read/write access. If you do not want it you can create a custom policy with just the logs part.

Adding the AWSLambdaExecute group solved my issue. What's weird is it didn't need that group initially to log, but only after redeploying the lambda. — C.M., Nov 11 '21 at 18:12

score 1 · Answer 7 · answered Sep 28 '16 at 17:40

Make sure you have the full path of your "Existing role" in your lambda function "Configuration":

Role: Choose an existing role Existing Role: service-role/yourRoleName

For some reason, typing only yourRoleName will work for some services (like SES) but not for CloudWatch.

Also, you may try creating a new role instead of using an existing one. This will create the role with the proper configuration (hopefully).

score 1 · Answer 8 · answered Jan 24 '18 at 19:56

For the issue was I was trying to create a log group in the Cloudformation script by : AWS::Logs::LogGroup and then trying to push the Lambda log to this log group. :P Novice After careful reading , i found that Lambda creates its own log with the aforementioned format: /aws/lambda/ We just need to provide policy permission to this log group , or just a generic permission with resource as: arn:aws:logs:::*

hope this helps

score 0 · Answer 9 · answered Dec 30 '17 at 14:52

Maybe a bit late, but for those who still struggle with seeing the lambda logs in cloudwatch. I noticed this regarding the lambda function's execution role: "You may use an existing role with this function. Note that the role must be assumable by Lambda and must have Cloudwatch Logs permissions." So in IAM i granted " CloudWatchLogsFullAccess" to the role i assigned to my function. then in cloudwatch, under logs, you'll see the logs for the functions assigned this role.

score 0 · Answer 10 · answered Dec 04 '18 at 09:43

It might already log, we just couldn't find the logs we expect...

e.g.

app.use(function simpleLogger (req, res, next) {
  console.info('[Logger]', req.method, req.originalUrl)
  next()
})

After performing GET /hello?world=1,

Local console: (simple and clear, nice!)

[Logger] GET /hello?world=1

CloudWatch Logs: (can you easily find the exact log below?)

START RequestId: a3552c34-f7a6-11e8-90ba-2fb886f31fb0 Version: $LATEST
2018-12-04T09:26:11.236Z  a3552c34-f7a6-11e8-90ba-2fb886f31fb0  [Logger] GET /hello?world=1
END RequestId: a3552c34-f7a6-11e8-90ba-2fb886f31fb0
REPORT RequestId: a3552c34-f7a6-11e8-90ba-2fb886f31fb0  Duration: 41.02 ms  Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 29 MB

Conclusion: too verbose to locate the original logs.

score 0 · Answer 11 · answered Jan 27 '19 at 21:29

0

I encountered this problem but none of the answers above solved my issue. It turns out that the region was somehow set to Ohio when I first started CloudWatch. After I changed it to US East (N. Virginia), everything works fine.

answered Jan 27 '19 at 21:29

randomcat

413
1
4
16

score 0 · Answer 12 · answered Jan 28 '19 at 03:43

0

CloudWatch & CloudWatch Logs are different Permissions, you need add CloudWatch Logs to the policy which attached with your role.

answered Jan 28 '19 at 03:43

Sherry Li

1

score 0 · Answer 13 · answered Jul 08 '19 at 21:56

0

There's a writeup called How to Monitor AWS Lambda with CloudWatch with a section on "How to Use CloudWatch Logs with Lambda". Looks like you already found your answer, but for anybody without the IAM specific issues, this may help.

answered Jul 08 '19 at 21:56

mbarlocker

1,310
10
16

score 0 · Answer 14 · answered Feb 07 '20 at 22:16

0

You might want add the Last Ingestion Time column to the log output. It's taking a few minutes for all my events to be written.

answered Feb 07 '20 at 22:16

C.M.

1,474
13
16

score 0 · Answer 15 · answered May 25 '20 at 17:33

Though this got answered already, Just wanted to add my experience which might be useful for others.

Even the permissions are set appropriately so that lambda can log in cloudwatch, sometimes it is taking 3-4 hrs to reflect the log groups.

In my case when using lambda for DynamoDB events, where lambda was given all needed permissions for both cloudwatch and DynamoDB. It took 4 hrs to get the logs reflected. May be some sync issues from AWS end. After 4 hrs without any action from my end, I was able to see the Cloudwatch logs.

Also make sure that you are seacrching the logs in the same region where your function is created.

score 0 · Answer 16 · answered Dec 03 '21 at 12:42

0

In my case, the permission boundary set on the role was the issue.

Although the role policy allowed logs:PutLogEvents, the permission boundary set on the role disallowed this action on the log group.

answered Dec 03 '21 at 12:42

Guy

1,254
17
16

score 0 · Answer 17 · edited May 23 '23 at 22:25

0

For me, it turned out to be that the cloudwatch arn that the IAM role should point to should be arn:aws:logs:<region>:***:log-group:/aws/lambda/<lambda-function-name>:* instead of arn:aws:logs:<region>:***:log-group:/aws/lambda/<lambda-function-name>

edited May 23 '23 at 22:25

Maria Ines Parnisari

16,584
9
85
130

answered May 04 '23 at 03:59

Suraj Sangani

1
1

Can't get AWS Lambda function to log (text output) to CloudWatch

17 Answers17

July 2020 Update !!

e.g.

Linked