I'm stumped on this one, i've spent the past ~3 days on it and done plenty of debugging/research attempts to no avail.
I have a docker container of Tomcat with a WAR deployed. This container works fine when started (independently or in docker-compose). However, when this container is deployed in a kubernetes cluster, Tomcat fails to deploy the WAR app. Due to the following exception:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceProxyEs' defined in ServletContext resource [/WEB-INF/index-clients_es_portal.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 100 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.elasticsearch.common.logging.log4j.Log4jESLoggerFactory.newInstance(Log4jESLoggerFactory.java:38)
at org.elasticsearch.common.logging.ESLoggerFactory.newInstance(ESLoggerFactory.java:82)
at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:66)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:121)
at org.elasticsearch.common.settings.Settings.<clinit>(Settings.java:63)
at com.esri.gw.index.elasticsearch.portal.ElasticSearchClient.init(ElasticSearchClient.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1702)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1641)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
... 107 more
I confirmed that log4j jar exists in WEB-INF/lib and is loaded by adding -verbose:class to tomcat JAVA_OPTS, this is outputted in catalina startup just before the above error:
[Loaded org.elasticsearch.common.logging.DeprecationLogger from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.elasticsearch.common.logging.Loggers from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.elasticsearch.common.Classes from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.elasticsearch.common.logging.ESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar][Loaded org.elasticsearch.com
mon.logging.jdk.JdkESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.elasticsearch.common.logging.log4j.Log4jESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.elasticsearch.common.logging.ESLogger from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/elasticsearch-2.3.2.jar]
[Loaded org.apache.log4j.spi.LoggingEvent from file:/home/root/apache-tomcat-7.0.72/lib/log4j-1.2.16.jar]
[Loaded org.springframework.core.NestedExceptionUtils from file:/home/root/apache-tomcat-7.0.72/webapps/app%23sharing/WEB-INF/lib/spring-core-4.1.6.RELEASE.jar]
[Loaded java.lang.Throwable$PrintStreamOrWriter from /home/root/java/jdk1.8.0_111/jre/lib/rt.jar]
[Loaded java.lang.Throwable$WrappedPrintWriter from /home/root/java/jdk1.8.0_111/jre/lib/rt.jar]
[Loaded java.util.IdentityHashMap$KeySet from /home/root/java/jdk1.8.0_111/jre/lib/rt.jar]
How to resolve this? And why does it occur only when container is deployed to k8s v1.9 which is running docker v1.2?
EDIT: this is not a duplicate of Copying files from host to Docker container
