I am using AI Platform training
to run ML training job using python
3.7.6
. I am using the abseil module for logging messages with absl-py
0.9.0
. I look at the instruction on how to direct python logging messages to stackdriver configuration medium article. I am using google-cloud-logging
1.15.0
. I did some very basic code to understand the issue with my configuration.
from absl import logging
from absl import flags
from absl import app
import logging as logger
import google.cloud.logging
import sys
import os
FLAGS = flags.FLAGS
def main(argv):
logging.get_absl_handler().python_handler.stream = sys.stdout
# Instantiates a client
client = google.cloud.logging.Client()
# Connects the logger to the root logging handler; by default this captures
# all logs at INFO level and higher
client.setup_logging()
fmt = "[%(levelname)s %(asctime)s %(filename)s:%(lineno)s] %(message)s"
formatter = logger.Formatter(fmt)
logging.get_absl_handler().setFormatter(formatter)
# set level of verbosity
logging.set_verbosity(logging.DEBUG)
print(' 0 print --- ')
logging.info(' 1 logging:')
logging.info(' 2 logging:')
print(' 3 print --- ')
logging.debug(' 4 logging-test-debug')
logging.info(' 5 logging-test-info')
logging.warning(' 6 logging-test-warning')
logging.error(' 7 logging test-error')
print(' 8 print --- ')
print(' 9 print --- ')
if __name__ == '__main__':
app.run(main)
First abseil send all logs to the stderr. Note sure if this is expected or not. In the screenshot below we see:
- print messages using
print
are display (later in the logfile fromStackdriver
) - Abseil logging messages appear 2 times. One with the right label in stack driver (DEBUG, INFO, WARNING or ERROR) and one more time with the special formatting
[%(levelname)s %(asctime)s %(filename)s:%(lineno)s] %(message)s
but always with the ERROR label inStackdriver
. - When I run the code locally I don't see duplicate.
Any idea how to have this setup properly to see the logging messages (using abseil) once with the proper "label" in Stackdriver
?
----- EDIT --------
- I am seeing the issue locally and not only when running on
GCP
. - The duplicate log messages appear when I add this line:
client.setup_logging()
. Before, I have no duplicate and all log messages are in the stdout stream - If I look at the logger
logger.root.manager.loggerDict.keys()
, I see a lot of them:
dict_keys(['absl', 'google.auth.transport._http_client', 'google.auth.transport', 'google.auth', 'google','google.auth._default', 'grpc._cython.cygrpc', 'grpc._cython', 'grpc', 'google.api_core.retry', 'google.api_core', 'google.auth.transport._mtls_helper', 'google.auth.transport.grpc', 'urllib3.util.retry', 'urllib3.util', 'urllib3', 'urllib3.connection', 'urllib3.response', 'urllib3.connectionpool', 'urllib3.poolmanager', 'urllib3.contrib.pyopenssl', 'urllib3.contrib', 'socks', 'requests', 'google.auth.transport.requests', 'grpc._common', 'grpc._channel', 'google.cloud.logging.handlers.transports.background_thread', 'google.cloud.logging.handlers.transports', 'google.cloud.logging.handlers', 'google.cloud.logging', 'google.cloud', 'google_auth_httplib2'])
- If I look at:
root_logger = logger.getLogger() for handler in root_logger.handlers: print("handler ", handler)
I see:
handler <ABSLHandler (NOTSET)>
handler <CloudLoggingHandler <stderr> (NOTSET)>
handler <StreamHandler <stderr> (NOTSET)>
and we can see that the stream is stderr and not stdout. I didn''t managed to change it.
I saw this discussion stackoverflow thread and I tried the last solution by @Andy Carlson but then all my logging message are gone.