I'm attempting to upgrade from the legacy fluentd logging agent to fluent-bit on Container-Optimized OS, as recommended here. These are Java logs in JSON format, produced by logback, and configured using com.google.cloud.spring.logging.StackdriverJsonLayout.
The instructions in the link above indicates that making the switch should be pretty painless, but with the setup above the resulting json data exported to stackdriver is in the wrong format, causing the logs to be pretty much useless when viewed in stackdriver console.
Specifically, the legacy fluentd agent configuration appears to understand the json format and just adds a couple of more attributes:
{
insertId: "abcder"
jsonPayload: {
cos.googleapis.com/container_name: "klt--kquu"
logger: "a.b.c.MyClass"
cos.googleapis.com/stream: "stdout"
thread: "pool-32-thread-8"
cos.googleapis.com/container_id: "xxxxx"
context: "default"
message: "Here's the message"
}
labels: {...}
logName: "something"
receiveTimestamp: "2023-05-29T11:07:33.771458482Z"
resource: {...}
severity: "INFO"
timestamp: "2023-05-29T11:07:27.282Z"
}
This format will neatly display "Here's the message" in the overview log view in stackdriver.
The default fluent-bit configuration however pushes the json data to a lower level, storing the entire json structure under the "message" attribute:
{
insertId: "abcde"
jsonPayload: {
cos.googleapis.com/container_id: "xxxxx"
time: "2023-05-29T11:41:32.662229708Z"
message: {
severity: "INFO"
timestampSeconds: 1685360492
context: "default"
timestampNanos: 661000000
logger: "a.b.c.MyClass"
thread: "pool-32-thread-5"
message: "Here's the message"
}
cos.googleapis.com/container_name: "klt--kquu"
cos.googleapis.com/stream: "stdout"
}
logName: "something"
receiveTimestamp: "2023-05-29T11:41:33.619576528Z"
resource: {...}
timestamp: "2023-05-29T11:41:32.662229708Z"
}
This will unfortunately mess up the default log view in stack driver; what I will see there is the full json string of everything under the message, starting with {cos.googleapis.com/container_id: xxxxx, cos.googleapis.com/container_name: klt--kquu, i.e. the container attribute under json payload are displayed first.
To me this feels like a bug: the default config for fluent-bit does not produce output that can be displayed properly in the GUI based on the same input that was working fine with fluentd.
Has anyone else run into this? Any ideas on how to work around it? Since everything was working out of the box before, I would very much like to avoid having to manually reconfigure the default COS fluent-bit configuration.
