0

I'm running a Docker container on a GCE instance under the Container-optimizes OS. I've followed this manual https://cloud.google.com/container-optimized-os/docs/how-to/logging and added the google-logging-enabled metadata value to enable sending containers' logs to Cloud Logging:

instance metadata screenshot

But it doesn't work. I can't see containers' logs in the Cloud Logging:

cloud logging screenshot

While I have a lot of container logs in reality:

container logs screenshot

How to make it working and really send containers' logs to Cloud Logging?

Update The log filter is the default one: (resource.type="gce_instance" AND resource.labels.instance_id="***") OR (resource.type="global" AND jsonPayload.instance.id="***") that means that all types of logs from the current instance should be shown.

  • Use this filter "resource.type="gce_instance" AND resource.labels.instance_id="xxxxxxxxx" AND logName=projects/xxxxxxxxxx/logs/cos_containers" – Mahboob Mar 04 '21 at 16:48
  • COS has a low logging level set. This means logs at a higher level will not be sent to Stackdriver. I did some research on this but never found a permanent solution to change the log level. I think only ERROR and WARN are passed up to Stackdriver. My answer on this post has details: https://stackoverflow.com/a/65768214/8016720 – John Hanley Mar 04 '21 at 21:51
  • @John Hanley, What's the relation between Cloud Logging and Stackdriver? As far as I understood they are two different services? –  Mar 05 '21 at 02:28
  • Stackdriver is the old name for `Operations: Cloud Logging`. I still use the term Stackdriver out of habit. One of my favorite services. – John Hanley Mar 05 '21 at 02:51

1 Answers1

1

My issue was that my service account associated with the instance had no the Logs Writer permission. I found that by running the sudo systemctl status stackdriver-logging command which shown the following:

● stackdriver-logging.service - Fluentd container for Stackdriver Logging
   Loaded: loaded (/usr/lib/systemd/system/stackdriver-logging.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2021-03-05 01:18:55 UTC; 1h 12min ago
 Main PID: 385 (docker)
    Tasks: 9 (limit: 2382)
   Memory: 31.9M
      CPU: 751ms
   CGroup: /system.slice/stackdriver-logging.service
           └─385 /usr/bin/docker run --rm --name=stackdriver-logging-agent -v /etc/stackdriver/logging.config.d/:/etc/google-fluentd/config.d/ -v /var/log:/var/log -v /var/lib/docker/containers/:/v

Mar 05 02:30:11 jwp-gitlab-runner-m3l0 docker[385]: 2021-03-05 02:30:11 +0000 [warn]: #0 Dropping 1 log message(s) error="User unauthorized to access *** for resource ***

By googling the error message I found the answer in the GitHub issue https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/issues/295

The following answer also helped me to diagnose the problem: https://stackoverflow.com/a/65768214/8016720

Now I have all the logs I want in Cloud Logging.