Using log4j2 in Spark java application

Question

I'm trying to use log4j2 logger in my Spark job. Essential requirement: log4j2 config is located outside classpath, so I need to specify its location explicitly. When I run my code directly within IDE without using spark-submit, log4j2 works well. However when I submit the same code to Spark cluster using spark-submit, it fails to find log42 configuration and falls back to default old log4j.

Launcher command

${SPARK_HOME}/bin/spark-submit \
--class my.app.JobDriver \  
--verbose \
--master 'local[*]' \
--files "log4j2.xml" \
--conf spark.executor.extraJavaOptions="-Dlog4j.configurationFile=log4j2.xml" \
--conf spark.driver.extraJavaOptions="-Dlog4j.configurationFile=log4j2.xml" \
myapp-SNAPSHOT.jar

Log4j2 dependencies in maven

<dependencies>
. . . 
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
        <!-- Bridge log4j to log4j2 -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-1.2-api</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
        <!-- Bridge slf4j to log4j2 -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
 <dependencies>

Any ideas what I could miss?

Vladimir Kroz · Answer 1 · 2017-05-02T23:15:45.607

10

Apparently at the moment there is no official support official for log4j2 in Spark. Here is detailed discussion on the subject: https://issues.apache.org/jira/browse/SPARK-6305

On practical side that means:

If you have access to Spark configs and jars and can modify them, you still can use log4j2 after manually adding log4j2 jars to SPARK_CLASSPATH, and providing log4j2 configuration file to Spark.
If you run on managed Spark cluster and have no access to Spark jars/configs, then you still can use log4j2, however its use will be limited to the code executed at driver side. Any code part running by executors will use Spark executors logger (which is old log4j)

edited May 02 '17 at 23:15

answered Mar 01 '17 at 16:46

Vladimir Kroz

5,237
6
39
50

Could you explain how to do so? I do have complete access to Spark and can modify configuration files, jars, etc. I downloaded some log4j2 jars and placed them on a "extra_jars" directory, and the referenced those on spark-env.sh using the classpath. however, Spark is till using old log4j for internal logs... – Ander Jan 11 '18 at 20:41
Just a remind, if log4j2 jar has been added to executor classpath, then log4j2 will be used anyway, so must add log4j2.xml file to executor. – Xilang Apr 30 '19 at 03:01

score 1 · Answer 2 · answered Feb 28 '17 at 06:12

Spark falls back to log4j because it probably cannot initialize logging system during startup (your application code is not added to classpath).

If you are permitted to place new files on your cluster nodes then create directory on all of them (for example /opt/spark_extras), place there all log4j2 jars and add two configuration options to spark-submit:

--conf spark.executor.extraClassPath=/opt/spark_extras/*
--conf spark.driver.extraClassPath=/opt/spark_extras/*

Then libraries will be added to classpath.

If you have no access to modify files on cluster you can try another approach. Add all log4j2 jars to spark-submit parameters using --jars. According to the documentation all these libries will be added to driver's and executor's classpath so it should work in the same way.

Yep, apparently this was the root cause: "Spark cannot initialize logging system properly". Few extra details on this: — Vladimir Kroz, Mar 01 '17 at 16:35

score 0 · Answer 3 · answered Mar 08 '17 at 10:22

0

Try using the --driver-java-options

${SPARK_HOME}/bin/spark-submit \    
--class my.app.JobDriver \      
--verbose \    
--master 'local[*]' \    
--files "log4j2.xml" \    
--driver-java-options "-Dlog4j.configuration=log4j2.xml" \    
--jars log4j-api-2.8.jar,log4j-core-2.8.jar,log4j-1.2-api-2.8.jar \    
myapp-SNAPSHOT.jar

answered Mar 08 '17 at 10:22

mgarzon

1

1

I think you are missing to add those jars into ExtraClassPath for both driver and executor – kensai Nov 20 '18 at 11:22

score 0 · Answer 4 · answered Mar 24 '18 at 03:34

If log4j2 is being used in one of your own dependencies, it's quite easy to bipass all configuration files and use programmatic configuration for one or two high level loggers IF and only IF the configuration file is not found.

The code below does the trick. Just name the logger to your top level logger.

private static boolean configured = false;

private static void buildLog()
{
    try
    {

        final LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
        System.out.println("Configuration found at "+ctx.getConfiguration().toString());

        if(ctx.getConfiguration().toString().contains(".config.DefaultConfiguration"))
        {

            System.out.println("\n\n\nNo log4j2 config available. Configuring programmatically\n\n");

            ConfigurationBuilder<BuiltConfiguration> builder = ConfigurationBuilderFactory
                    .newConfigurationBuilder();

            builder.setStatusLevel(Level.ERROR);
            builder.setConfigurationName("IkodaLogBuilder");

            AppenderComponentBuilder appenderBuilder = builder.newAppender("Stdout", "CONSOLE")
                    .addAttribute("target", ConsoleAppender.Target.SYSTEM_OUT);
            appenderBuilder.add(builder.newLayout("PatternLayout").addAttribute("pattern",
                    "%d [%t]  %msg%n%throwable"));
            builder.add(appenderBuilder);               

            LayoutComponentBuilder layoutBuilder = builder.newLayout("PatternLayout").addAttribute("pattern",
                    "%d [%t] %-5level: %msg%n");

            appenderBuilder = builder.newAppender("file", "File").addAttribute("fileName", "./logs/ikoda.log")
                    .add(layoutBuilder);
            builder.add(appenderBuilder);


            builder.add(builder.newLogger("ikoda", Level.DEBUG)
                    .add(builder.newAppenderRef("file"))
                    .add(builder.newAppenderRef("Stdout"))
                    .addAttribute("additivity", false));

            builder.add(builder.newRootLogger(Level.DEBUG)
                    .add(builder.newAppenderRef("file"))
                    .add(builder.newAppenderRef("Stdout")));
            ((org.apache.logging.log4j.core.LoggerContext) LogManager.getContext(false)).start(builder.build());
            ctx.updateLoggers();
        }
        else
        {
            System.out.println("Configuration file found.");
        }
        configured=true;
    }
    catch(Exception e)
    {
        System.out.println("\n\n\n\nFAILED TO CONFIGURE LOG4J2"+e.getMessage());
        configured=true;
    }
}

Using log4j2 in Spark java application

4 Answers4

Linked