Label logging with python to gcp log viewer

Question

I am trying to use GCP labels in my GKE app, I would like to log labels that are associated with the GKE like namespace_name,location, cluster_name, project_id and pod_name, however by default everything is logged under global resource type and only project_id is populated in GCP Log Viewer

Example:

I found a way to to change resource type to k8s_container and populate other labels by using Resource, however I can't seem to find a way to populate labels without hardcoding values directly in to the app, see example below:

handler = CloudLoggingHandler(client,resource=Resource("k8s_container",labels={

'namespace_name': "my_namespace",
'location': "us-west1",
'cluster_name': "my-cluster",
'project_id': "my-project-id",
'pod_name': "my-pod"
}))

---

Full app for the reference:

import logging
import google.cloud.logging
from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging.resource import Resource

client = google.cloud.logging.Client().from_service_account_json(GOOGLE_SA_ACCOUNT)

handler = CloudLoggingHandler(client,resource=Resource("k8s_container",labels={

    'namespace_name': "my_namespace",
    'location': "us-west1",
    'cluster_name': "my-cluster",
    'project_id': "my-project-id",
    'pod_name': "my-pod"
}))

google.cloud.logging.handlers.setup_logging(handler)

logging.getLogger().setLevel(logging.ERROR)


try:
    requests.head('https://www.exampleurladdress.com').status_code == 200
except requests.exceptions.RequestException as err:
    logging.error(err

I was wondering if there is some sort of a way to populate them properly directly from GKE ?

Thanks.

Answer 1

you can populate the resource field properly if you print your log payloads to stdout. The logging agent that is present at each worker node of a GKE cluster will parse each line printed to stdout and will ingest the parsed payload as a separate log entry with the correct resource field values.

Consider to use the v3.0.0 of Logging client library for Python. It provides a support for printing out a stringified Json in the format that the logging agent knows to parse. This way beside the payload, you can ingest user's labels, HTTP request context and other metadata. It also allows you to ingest the payload as a text or Json structure.