how to add custom field to log logback-logstash

Question

My spring boot log currently looks like the following.

{"@timestamp":"2018-08-07T14:49:21.244+01:00","@version":"1","message":"Starting Application on ipkiss bla bla)","logger_name":"logger name....","thread_name":"main","level":"INFO","level_value":20000}

with the logback-spring.xml setup like below

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.ipkiss.correlate.logback.CorrelationPatternLayoutEncoder">
            <pattern>%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} id = %id  %m%n%wEx</pattern>
        </encoder>
</appender>

and my class for LayoutEncoder looks like this

public class CorrelationPatternLayoutEncoder extends PatternLayoutEncoder {
    public CorrelationPatternLayoutEncoder() {

    }

@Override
public void start() {
    PatternLayout patternLayout = new PatternLayout();
    patternLayout.getDefaultConverterMap().put("id", CorrelationConverter.class.getName());
    patternLayout.setContext(context);
    patternLayout.setPattern(getPattern());
    patternLayout.setOutputPatternAsHeader(outputPatternAsHeader);
    patternLayout.start();
    this.layout = patternLayout;
    this.started = true;
}

}

what I was trying to achieve is to add the id to the log, I can't make logstach append my id, i tried Custom field according to the docs but I couldn't make it work, any ideas how I can achieve this?

this is what i want to end up with

{"id":"3a7ccd34-d66a-4fcc-a12e-763a395a496c","@timestamp":"2018-08-07T14:49:21.244+01:00","@version":"1","message":"Starting Application on ipkiss bla bla)","logger_name":"logger name....","thread_name":"main","level":"INFO","level_value":20000}

or id being appended at the end of the log.

score 9 · Answer 1 · edited May 28 '20 at 17:42

From the logstash-encoder github page

By default, each entry in the Mapped Diagnostic Context (MDC) (org.slf4j.MDC) will appear as a field in the LoggingEvent.

So in short, if you add your id entry into MDC it will automatically be included in all of your logs.

To add your id to MDC do the following:

MDC.put("id", uuid);

As MDC is a thread local variable you will have to clear it after your request has finished using

MDC.remove("id")

In a web application, add and clearing the values in MDC is usually done in a servlet filter ie.

public class IdFilter implements Filter{
    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
        MDC.put("id", UUID.randomUUID().toString());
        try {
            filterChain.doFilter(servletRequest, servletResponse);
        } finally {
            MDC.remove("id");
        }
    }
}

the put method is put(String, String). You can not pass an Object. MDC is used for logging purposes so I suppose that is why they limited the method parameter to a String. You should convert your Object into whatever String representation suites your needs and then add that to MDC. Does this make sense? — Michael McFadyen, Feb 07 '19 at 11:50

score 3 · Answer 2 · answered Oct 17 '20 at 07:53

You can add custom log field by Creating a custom conversion specifier. First off all create a custom converter class which extends ClassicConverter class.

import ch.qos.logback.classic.pattern.ClassicConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;
import java.util.UUID;

public class LogUniqueIdConverter extends ClassicConverter {

    @Override
    public String convert(ILoggingEvent event) {
        return String.valueOf(UUID.randomUUID());
    }
}

ClassicConverter objects are responsible for extracting information out of ILoggingEvent instances and producing a String.

So just write in your logback configuration file so that logback know about the new Converter. For this you just need to declare the new conversion word in the configuration file, as shown below:

<configuration>

  <conversionRule conversionWord="customId" 
                  converterClass="com.naib.util.MySampleConverter" />
        
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>{"id": "%customId" %msg%n%throwable %msg%n}</pattern>
    </encoder>
  </appender>

  <root level="DEBUG">
    <appender-ref ref="STDOUT" />
  </root>
</configuration>

I like this approach more as the one using MDC as it doesn't throw an exception if the value is not set. At least that happened to me when not using a custom encoder — Johannes Klug, Apr 28 '21 at 09:29
This is missing the right class name, right? I guess `com.naib.util.MySampleConverter` should be `com.naib.util.LogUniqueIdConverter` for it to make sense. — oligofren, Aug 09 '21 at 11:23
Additional to this really good answer, in case you use IAccessEvent you may need to create a new class extending *AccessConverter* and overriding the convert method. — Mauricio Zárate, Nov 01 '21 at 04:42

how to add custom field to log logback-logstash

2 Answers2