Can't show info level logging in AWS Lambda

Question

I'm trying to run a script in AWS lambda and I want to output info level logs to the console after the script runs. I've tried looking for help from This post on using logs in lambda buy haven't had any success. I think AWS Cloudwatch is overriding my configuration shown bellow.


import logging
# log configuration
logging.basicConfig(
    format='%(levelname)s: %(message)s',
    level=logging.INFO,
    encoding="utf-8"
    )

I want to set the logging level to logging.info. How can I do this? The runtime is python 3.9

Yes, thank you, that fixed the problem. Now does anyone know why that worked and ```level=logging.INFO``` didn't? Once we know why I'll mark this closed. — David Chidester, Sep 28 '21 at 19:37
from my understanding, I think `logging.basicConfig(level=...)` affects the minimum log level at which logs show up in the console, but across *all* loggers. The one I added above explicitly sets the minimum enabled level for the root logger, i.e. `logging.getLogger()`. The enabled level for each logger determines at what level messages will be logged - otherwise, each call to a logger method like `logging.info` is basically a no-op. — rv.kvetch, Sep 28 '21 at 19:40
Could you post the above as an answer so I can mark it closed? — David Chidester, Sep 28 '21 at 19:46

score 5 · Accepted Answer · answered Sep 28 '21 at 19:56

from my understanding, I think one fix would be to add this:

logging.getLogger().setLevel('INFO')

I believe that logging.basicConfig(level=...) affects the minimum log level at which logs show up in the console, but across all loggers. The one above explicitly sets the minimum enabled level for the root logger, i.e. logging.getLogger(). The enabled level for each logger determines at what level messages will be logged - otherwise, each call to a logger method like logging.info is basically a no-op.

So essentially, the basicConfig and setLevel calls are separate, but they work together to determine if a library's logs are printed to the console. For example, you can set basicConfig(level='DEBUG') so that the debug-level logs for all libraries should get printed out. But if you want to make an exception for one library like botocore for example, you can use logging.getLogger('botocore').setLevel('WARNING'), and that will set the minimum enabled level for that library to WARNING, so only messages logged by this library above that minimum level get printed out to the console.

score 1 · Answer 2 · answered Sep 29 '21 at 05:26

You actually need to know that Python Logging in lambdas is a bit odd - if you are using the default logging module, when a lambda is invoked and it begins to spin up its back end, it creates a logging handler there and attaches it to the name of the log group as the handler name - not the name attribute as basicConfig does by default when it creates a logging handler

As such, basicConfig, attempting to modify the handler based on the name attribute, does NOT find the handler created by the lambda invoke start up and so will not work to update your settings, where as in your top level lambda handler file, as it can see the one logging handler at its level (in the lambda) when you use getLogger and setLevel you set that handler yourself

Therefor, if you just use getLogger() it works at the top most level (lambda handler and its file) because the lambda_handler is being imported into the backend code to run, so it can find the handler.

Any further imports however will be looking for the name path of the lambda_handler, and attaching their logging handler to that name path - meaning their logging statements will NOT show up in cloud watch.

There are three solutions I have found:

use logger = getLogger() and setLevel in the lambda handler and its file - any further files you import, do NOT use getLogger - instead just use import logging - logging.INFO(message) to force the logger to look for a default handler and use that (note: This is not idea, you end up loosing a lot of control over your log files
if you cannot use any additional libraries, then you have to write some code in your lambda handler to see if a current logging handler exists - if it does, grab that and adjust it as needed to populate down the rest of your imports. You can find code for that scattered around SO
if you can (and I generally dislike answers that say use this library, so thats why this is answer three even though I love this library) use aws_lambda_powertools and its module Logger - this is a very powerful logging module designed to work with your existing logging statements and aws handlers in python - and its an open source python project by aws themselves. Its got a lot of great tools in it besides just the logger, but the logger is very very awesome.

Can't show info level logging in AWS Lambda

2 Answers2