0

I have a container running in GKE which hosts a scala application. Across our stack, we're trying to improve debugging, and I'm trying to get our deployments to output structured logs. I've already achieved this for the python parts of our stack, but I'm having trouble with scala.

In scala we are currently using slf4j and logback to log messages. I've already done some configuration work to have logback format the output as single line json, but now I'm struggling to actually get it to pick up on custom fields.

In python I can just do something like this:

custom_fields = { "userID": "andy" }
logger.info("User logged in", custom_fields)

I've found a few examples of how to do it in scala, but I don't think they're suitable:

Here is an example using MDC, my concerns are that I don't think this is thread safe? I have multiple threads running asynchronously that are processing multiple users at once. I could wrap all my log calls in a mutex, and add/log/remove the field in the mutex, but this feels very heavy handed and like it could mess up the performance of the system: https://gquintana.github.io/2017/12/01/Structured-logging-with-SL-FJ-and-Logback.html

I've also seen examples of passing StructuredArgument to the log call: https://github.com/logstash/logstash-logback-encoder#event-specific-custom-fields

But as detailed in my previous question: How to import StructuredArgument for structured logging in scala using slf4j and logback

I cannot seem to get this working in Scala.

What's the best way to go about implementing this?

Hasanul Murad
  • 127
  • 6
Andy
  • 3,228
  • 8
  • 40
  • 65
  • If you are looking for ways to read the Stackdriver logs (GCP), you can use the client [libraries](https://cloud.google.com/logging/docs/setup#reading_logs) to call the logging API or Logging REST API endpoint. – Mahboob Aug 19 '20 at 23:00

1 Answers1

0

Yes, MDC and NDC in slf4j, logback, and other classic JVM logging systems use threadlocals so they can only be used in a single thread without struggling. It is just not designed for async logging.

But it is not so heavy as you think, because logback-slf4j MDC uses LogbackMDCAdapter, which is thread-safe but works with thread local semantic.

This means that you can pass a custom case class that contains the desired logging context, and then on each log, you will add this to MDC, log the initial message, and then clear MDC.

The helper function for this would be like this:

def asyncLog(msg: String, context: CustomLoggingContext){ 
  MDC.put(context)
  logger.log(msg)
  MDC.clear()
}

Other options would be:

  • Still use slf4j and write context info just as part of the plain message and then parse it in somewhere else (like in logstash).
  • Use another logging framework/system that has such functionality
  • Write your own
Artem Sokolov
  • 810
  • 4
  • 8