1

I am having some issues with the log entries in Stackdriver using GKE, when the log entry is greater than 20 KB, this is split into several chunks. According to GCP documentation, the limit size of log entries is 256 KB (https://cloud.google.com/logging/quotas). I have tested several configurations and found out something very curious: when the Cluster is set up using Ubuntu nodes thet issue is seen. When I use the default node type: Container-Optimized OS (cos), Stackdriver captures the log entries correctly.

Can somebody explain me the cause of this error?. I have checked this Logging with Docker and Kubernetes. Logs more than 16k split up, I think it could be related.

Additional information:

  • GKE static version: v1.14.10-gke.50
  • Kernel version (nodes): 4.15.0-1069-gke
  • OS image (nodes): Ubuntu 18.04.5 LTS
  • Docker version (nodes): 18.9.7
  • Cloud Operations for GKE: Legacy Logging and Monitoring

New feedback: I have created more clusters using different GKE versions and another "Cloud Operations for GKE" implementation (System and Workload and Monitoring) and the issue is the same one. Curret steps to reproduce the issue:

  • Create a GKE cluster using as image Ubuntu (No matter the GKE version)
  • Deploy an application which logs a log entry greater than 16 KB. I am using a Spring boot application + Log4j 1.X
  • Look for the log entry in the Stackdriver web console. The log entry is split into multiple chunks.
Carlos Andrés Holguín
  • 19
  • 2
  • 1
    Hello. Could you tell how you've managed to tell that the splitting is occurring? Can you provide the steps you followed that could be replicated to see this splitting? Do I assume correctly that you checked on exact the same versions only the image of OS is different? – Dawid Kruk Oct 15 '20 at 10:12
  • Also, please tell which monitoring solution are you using with this cluster? Is it the `Legacy Logging and Monitoring` (support is ending with `GKE` 1.14) or `Cloud Operations for GKE`? You can check it by following this link: https://cloud.google.com/stackdriver/docs/solutions/gke/installing#migrating – Dawid Kruk Oct 15 '20 at 13:45
  • Is there any particular reason why you are using `Ubuntu` image instead of `ContainerOS`? – Dawid Kruk Oct 15 '20 at 15:58
  • Hi @DawidKruk!. I am using Ubuntu because it is customer restriction. They have deployed and tested several applications in a cluster with this configuration (Ubuntu + Docker). Regarding the steps to replicated the issue and the monitoring solution, I will add more information – Carlos Andrés Holguín Oct 16 '20 at 17:02
  • What is the configuration of buffer for Log4j? – Serhii Rohoza Oct 17 '20 at 15:08
  • Hi @SerhiiRohoza!. I am using a default configuration, I am not setting any specific configuration related to buffer .This issue is also happening when I am sending something directly to the stdout, so the logger configuration is not an issue. – Carlos Andrés Holguín Oct 21 '20 at 19:39
  • Thank you. Why do you think it's an issue if you didn't configure logging buffers to service big logging entries? – Serhii Rohoza Oct 21 '20 at 21:34

1 Answers1

0

I see a similar happens in my GCP project when output of one log entry is large, (17KB). what difference is: the first log entry contains 0~40% of the full log output, the second contains 0~70%, and third/last contains 0~100%. My app is Spring Boot reactive application. I use a reactive log filter.

Chen Wang
  • 11
  • 1