java - jboss logmanager breaks json format of my application json-logs from log4j2 by escaping quotes

Question

Context is:

I have switched from logging to file as text towards logging to stdout as json, to send the log messages as parsable json-logs to the elastic stack with kibana.

The wildfly server (version 22.0.1.Final or 26.1.3.Final) uses the jboss logger which I also switched from Textbased logging to structured parsable json logs writing them also to stdout.

The jboss logger takes the logs from our java application (with log4j2) and wraps them into it's own log message enriching it with more context.

Problem is:

the application json-logs are getting escaped by the jboss logger before appending it, therefore breaking the json format and making it unparsable for the rest of our workflow.

This is the result with the setting jboss-logger in json format & log4j2 in json format - see the key 'message' where quotation marks are escaped like \"log.level\": \"INFO\" instead of "log.level": "INFO"

{
    "timestamp": "2023-04-26T09:01:50.489+02:00",
    "sequence": 295,
    "loggerClassName": "org.jboss.logmanager.Logger",
    "loggerName": "stdout",
    "level": "INFO",
    "message": "{\"@timestamp\":\"2023-04-25T12:01:50.488Z\",\"log.level\": \"INFO\",\"message\":\"creating a new account for userid 146\", \"ecs.version\": \"1.2.0\",\"service.name\":\"my_project_x\",\"service.version\":\"5.67\",\"event.dataset\":\"my_project_x.log\",\"process.thread.name\":\"default task-2\",\"log.logger\":\"org.my_project_x.logic.AccountImpl\",\"hostname\":\"Lenovo\",\"environment\":\"develop\",\"user.id\":\"146\",\"log\":{\"origin\":{\"file\":{\"name\":\"AccountImpl.java\",\"line\":720},\"function\":\"addAccount\"}}}",
    "threadName": "default task-2",
    "threadId": 183,
    "mdc": {},
    "ndc": "",
    "hostName": "lenovo-p14s",
    "processName": "jboss-modules.jar",
    "processId": 64672
}

then I tried turning the jboss logger back to Text format while leaving the application logs as json in log4j2. Like this the application logs stay correctly in the json format:

2023-04-26 10:49:59,407 INFO  [stdout] (default task-6) {"@timestamp":"2023-04-26T10:49:59,407Z","log.level": "INFO","message":"creating a new account for userid 166.", "ecs.version": "1.2.0","service.name":"my_project_x","service.version":"5.67","event.dataset":"my_project_x.log","process.thread.name":"default task-6","log.logger":"org.my_project_x.logic.AccountImpl","hostname":"Lenovo","environment":"develop","user.id":"166","log":{"origin":{"file":{"name":"AccountImpl.java","line":720},"function":"addAccount"}}}

but I would like to have the jboss logs also in the json format, while leaving my application logs in valid json format.

Question is:

I could not find any way to disable the escaping of the application logs when turning on the json format for the jboss logger. Is there any way to have our application logs in correct json format, while also having the jboss logs in json, resulting in a valid json (application log4j2) in json (jboss log).

Any help will be appreciated.

Regards,

ps: my setup if needed

for wildfly / jboss logger

<subsystem xmlns="urn:jboss:domain:logging:8.0">
    <console-handler name="CONSOLE">
        <level name="INFO"/>
        <formatter>
            <named-formatter name="JSON"/>
        </formatter>
    </console-handler>
    <periodic-rotating-file-handler name="FILE" autoflush="true">
        <formatter>
            <named-formatter name="PATTERN"/>
        </formatter>
        <file relative-to="jboss.server.log.dir" path="server.log"/>
        <suffix value=".yyyy-MM-dd"/>
        <append value="true"/>
    </periodic-rotating-file-handler>
    <logger category="com.arjuna">
        <level name="WARN"/>
    </logger>
    <logger category="io.jaegertracing.Configuration">
        <level name="WARN"/>
    </logger>
    <logger category="org.jboss.as.config">
        <level name="DEBUG"/>
    </logger>
    <logger category="sun.rmi">
        <level name="WARN"/>
    </logger>
    <root-logger>
        <level name="INFO"/>
        <handlers>
            <handler name="CONSOLE"/>
            <handler name="FILE"/>
        </handlers>
    </root-logger>
    <formatter name="PATTERN">
        <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
    </formatter>
    <formatter name="COLOR-PATTERN">
        <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
    </formatter>
    <formatter name="JSON">
        <json-formatter/>
    </formatter>
</subsystem>

for log4j2 (version 2.19.0) in the java application:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.my_project_x.logging" status="WARN" strict="true" verbose="true">
    <Appenders>
        <Console name="myConsoleLog">
            <IgnoreExceptions>false</IgnoreExceptions>
            <EcsLayout serviceName="my_project_x" serviceVersion="${serverVersion}" includeOrigin="true" eventDataset="my_project_x.log">
                <KeyValuePair key="hostname" value="%hostname"/>
                <KeyValuePair key="environment" value="develop"/>
                <KeyValuePair key="user.id" value="%userId"/>
            </EcsLayout>
        </Console>
    </Appenders>

    <Loggers>
        <Logger name="org.my_project_x">
            <Level>INFO</Level>
        </Logger>
        
        <Root>
            <AppenderRef ref="myConsoleLog"/>
            <Level>WARN</Level>
        </Root>
    </Loggers>
</Configuration>

Answer 1

You're using both the logging configuration in WildFly and including a logging configuration in your deployment with a handler which is writing to System.out. WildFly wraps System.out and System.err in a logger. When another log manager writes to one of the standard streams, then it's processed through the log manager.

With JSON logging you've got three options.

1. You can accept the JSON output that WildFly outputs and remove your logging configuration from your deployment. Note that this will be the most consistent output and include logs from WildFly.
2. You can set the direct configuration parameter to true. Do note the remarks there of how messages could still be interwined.
3. Write your JSON output to a file instead of the console.

Do note with the second option you may end up with two different JSON formats.