5

I am using Docker version 17.12.1-ce Kubernetes verision v1.10.11

My application prints the log in Json format to the console. One of the fields is stackTrace, which can include a huge stackTrace.

The problem is that the log message is split up into two messages. So if I look at the /var/lib/docker/containers/ ... .log I see two messages. I read that this is done for security reasons, but I don't really understand what I can do with that?

Should I cut my stackTrace? Or customize the size? Is this permitted? Is it the correct way to deal with this issue?

p/s I am using json-file logging driver

liotur
  • 809
  • 2
  • 17
  • 36

2 Answers2

3

This is an expected behavior. Docker chunks log messages at 16K, because of a 16K buffer for log messages. If a message length exceeds 16K, it should be split by the json file logger and merged at the endpoint.

It does mark the log as a partial message but really depends on the driver/service to re-assemble.

Docker Docs mentions that there are different supported drivers.

With your architecture (Stacktraces), the json-driver might be not a best option.

And I've found this thread on github that adds additional info on topic (as well as a lot of offtop).

Edit.

Logging Architecture says that everything a containerized application writes to stdout and stderr is handled and redirected somewhere by a container engine.

The Docker container engine redirects those two streams to a logging driver, which is configured in Kubernetes to write to a file in json format.

Note: The Docker json logging driver treats each line as a separate message. Another peculiarity is that when using the Docker logging driver, there is no direct support for multi-line messages. You need to handle multi-line messages at the logging agent level or higher.

I don't really understand what I can do with that?

It's a limitation on Docker size. Here is another good discussion that ends up with the idea to use filebeat/fluentd .

It looks like the Docker_mode option for Fluentbit might help, but I'm not sure how exactly you are parsing container logs.

Should I cut my stackTrace?

It depends if you need traces in logs or not.

Or customize the size? I have searched for some kind of "knob" to adjust on Docker side, and can't find any as of now.

It looks like the only solution for that is to use some log processing tool that can combine split lines.

Nick
  • 1,882
  • 11
  • 16
2

Log lines are limited to 16k per log line by container runtime. In case logs from your app exceed this limit, the logs are going to be split across multiple lines.

You have two options

  • Either concat the logs when they are being read by some log collector/processor like fluentd from /var/lib/docker/containers/*.log
  • OR Concat it in the log aggregator tool(splunk, ELK) while reading the logs. (This is not very efficient since this concat operation has to happen every time the logs are read)

Lets take below example. assume that the log is very large which results in split into two lines. This results in some part of the log in Line 1(first 16k) and remaining in Line 2. And also now the json format is broken.

2022-05-26 00:36:53,066 INFO job.logger - {"queryId":"1d71345c-d095-c50f-3ed4-52bbf6073100","context":"[system]","queryText":"SELECT * FROM schemata","setupTime":0,"waitTime":0,"start":1653525410464,
"end":1653525410499,"memoryAllocated":0} job.logger.EOL

Note that having some specific keyword at the end of the line (from above eg: "job.logger.EOL") will make it easy for you to stitch those two lines back together either in log collector or log aggregator.

In case you are using fluentd, you can use concat which will stitch those two line together before shipping it to the log aggregator.

<filter docker.log>
  @type concat
  key loga
  separator ""
  multiline_start_regexp /job.logger/
  multiline_end_regexp /job.logger.EOL/
</filter>
ns15
  • 5,604
  • 47
  • 51