Spark unable to load custom log4j.properties from fat-jar resources

Question

I'm trying out new and improved spark 1.4.1 but I experience what looks like a regression.

When spark-submit my jar, I pass along the instruction to load a custom log4j-light-redis.properties bundled in the resources of my fat-jar. In spark 1.3.1, this works like a charm. In 1.4.1, resource loader does not find it:

log4j: Trying to find [log4j-light-redis.properties] using context classloader sun.misc.Launcher$AppClassLoader@4e25154f.
log4j: Trying to find [log4j-light-redis.properties] using sun.misc.Launcher$AppClassLoader@4e25154f class loader.
log4j: Trying to find [log4j-light-redis.properties] using ClassLoader.getSystemResource().
log4j: Could not find resource: [log4j-light-redis.properties].

In spark 1.3.1, it would find the resource correctly:

log4j: Trying to find [log4j-light-redis.properties] using context classloader org.apache.spark.util.MutableURLClassLoader@1a1d6a08.
log4j: Using URL [jar:file:/E:/Projects/.../target/fatjar-1.1-SNAPSHOT-shaded.jar!/log4j-light-redis.properties] for automatic log4j configuration.
log4j: Reading configuration from URL jar:file:/E:/Projects/.../target/fatjar-1.1-SNAPSHOT-shaded.jar!/log4j-light-redis.properties
log4j: Parsing for [root] with value=[WARN,console,redis].

Someone also had the same issue but spark mailing list still haven't acknowledged the question:

http://apache-spark-user-list.1001560.n3.nabble.com/log4j-xml-bundled-in-jar-vs-log4-properties-in-spark-conf-tt23923.html

Update Looks like it's event worse than what I thought.. I'm not able to load a custom log4j appender from my fat-jar anymore! It fails with a ClassNotFoundException!

Answer 1

In 1.4.1, MutableURLClassLoader is not set before log4j initialization like it is in 1.3.1.

Here is the explanation:

While parsing arguments in in SparkSubmit.scala, it uses spark.util.Utils. This object has a new static dependency on log4j, through ShutdownHookManager, that triggers it's initialization before the call to setContextClassLoader(MutableURLClassLoader) from submit > doRunMain > runMain