I created a handler for the root logger and was expecting all the messages that I see on the terminal, including not from my own library, to be captured. However, the logs from external API calls do not get captured.
I have a setup with a web service which exposes an API to create/monitor our ML models. The service runs in a docker container. My goal is to capture and redirect logs into appropriate streams into CloudWatch. An important detail is the destination (Cloudwatch) log stream varies depending on the api call parameters.
I am adding a handler to the root logger which send the logs into the desired destination on CloudWatch. My own logs work fine, but the SageMaker logs do not go there. A simplified version looks like this:
import logging
from watchtower import CloudWatchLogHandler
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
def sagemaker_api_call():
# this is a stub for an arbitrary sagemaker API call which produces logs
pass
def api_call(job_id: str):
model_name = 'test_model'
handler=CloudWatchLogHandler(
log_group_name=f'/ml/{model_name}',
log_stream_name=job_id,
use_queues=False,
log_group_retention_days=30)
logger.addHandler(handler) # this works, I can see the message in Cloudwatch
logger.info('Log started')
sagemaker_api_call() # this does not send logs to my stream in Cloudwatch
Now when I am running this in docker, I can see the logs from SageMaker being printed into the terminal once sagemaker_api_call gets called. They also DO get logged into CloudWatch into the destinations which SageMaker defines. However, they do not get logged in my stream, which is what I want.
I also tried adding the same handler to the logger called 'sagemaker' (this is where most of the sagemaker modules write to), no help. What am I doing wrong?
