Spark not taking log4j2.xml

Question

I am trying to use log4j2.xml instead of the default log4j logging of spark.

My Log4j2.xml is as below

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE log4j:configuration PUBLIC
  "-//APACHE//DTD LOG4J 1.2//EN" "http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/doc-files/log4j.dtd">

<Configuration status="WARN" name="MyApp" monitorInterval="30">

        <Properties>
            <Property name="appName">MyApp</Property>
            <Property name="appenderPatternLayout">%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n</Property>
            <Property name="fileName">/app/vodip/logs/${appName}.log</Property>
        </Properties>

        <Appenders>
            <RollingFile name="RollingFile"
                         fileName="${fileName}"
                         filePattern="a1
                         ${appName}-%d{yyyy-MM-dd-HH}-%i.log">
                <PatternLayout>
                    <Pattern>${appenderPatternLayout}</Pattern>
                </PatternLayout>
                <Policies>
                    <TimeBasedTriggeringPolicy interval="4" modulate="true"/>
                    <SizeBasedTriggeringPolicy size="250 MB"/>
                </Policies>
            </RollingFile>
        </Appenders>


      <Loggers>
          <Logger name="xyz.abcs.MyApp" level="debug" additivity="false">
              <AppenderRef ref="RollingFile"/>
          </Logger>
          <Root level="debug">
              <AppenderRef ref="RollingFile"/>
          </Root>
      </Loggers>

    </Configuration>

I have places my log4j2.xml in spark/conf folder on all nodes and restarted spark and submitted spark programs as below.

spark-submit --master spark://xyzzz.net:7077 \
--class abcd.myclass \
--deploy-mode cluster --executor-memory 2G --total-executor-cores 4  \
--conf spark.network.timeout=150 \
--files /app/spark/spark-1.6.1-bin-hadoop2.6/conf/log4j2.xml \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j2.xml" \
--driver-java-options "-Dlog4j.configuration=file:/app/spark/spark-1.6.1-bin-hadoop2.6/conf/log4j2.xml" \
/app/spark/my.jar

I a seeing this in my worker stderr log. It means that my logs are not using log4j2 functionality.

log4j:WARN Continuable parsing error 10 and column 78 log4j:WARN Document root element "Configuration", must match DOCTYPE root "null". log4j:WARN Continuable parsing error 10 and column 78 log4j:WARN Document is invalid: no grammar found. log4j:ERROR DOM element is - not a element. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Can anyone advise what is wrong with configuration???

I have added this line in code. But it gave log4j:WARN Continuable parsing error 13 and column 78 log4j:WARN Document root element "Configuration", must match DOCTYPE root "log4j:configuration". log4j:WARN Continuable parsing error 13 and column 78 log4j:WARN Element type "Configuration" must be declared. log4j:WARN Continuable parsing error 15 and column 17 — AKC, Oct 19 '16 at 17:46
seems log4j2 is not friend with spark check [this](https://issues.apache.org/jira/browse/SPARK-6305?jql=project%20%3D%20Spark%20AND%20text%20~%20log4j2) and [this](https://github.com/apache/spark/pull/4998#issuecomment-80944189) if you have not already — VladoDemcak, Oct 19 '16 at 18:33

score 2 · Answer 1 · answered Feb 27 '17 at 22:24

2

At least one mistake in your command line can lead to this error

-Dlog4j.configuration=. . . must actually be -Dlog4j.configurationFile=. . . when using log4j2

log4j.configuration is parsed by old log4j, which obviously don't understand new configuration format and throws parsing errors

answered Feb 27 '17 at 22:24

Vladimir Kroz

5,237
6
39
50

thanks, i am trying from 1 day to sort this out. – Karthik Jul 26 '23 at 02:43

score 0 · Answer 2 · answered Oct 19 '16 at 17:31

are you also putting log4j file in resource folder of your project if it will be there then remove it from there and for logging spark application with log4j for driver and executer you should also provide path of log4j for driver and executor for that as follows

spark-submit --class MAIN_CLASS --driver-java-options "-Dlog4j.configuration=file:PATH_OF_LOG4J" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:PATH_OF_LOG4J" --master MASTER_IP:PORT JAR_PATH

You can also refer to this blog for more details https://blog.knoldus.com/2016/02/23/logging-spark-application-on-standalone-cluster/

I tried this but continued receiving error & warnings log4j:WARN Continuable parsing error 42 and column 43 log4j:WARN Element type "AppenderRef" must be declared. log4j:ERROR DOM element is - not a element. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties — AKC, Oct 19 '16 at 18:24

Spark not taking log4j2.xml

2 Answers2