7

We have a web app that we run in Tomcat 8, and recently we've observed that the artifacts (.war files) built by some developers on our team throw a NoClassDefFoundError, while the same code built by others functions as expected.

From logs/localhost.2018-05-11.log: 

org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: Could not initialize class org.geotools.referencing.datum.DefaultEllipsoid
    ...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.geotools.referencing.datum.DefaultEllipsoid
    at org.geotools.referencing.GeodeticCalculator.<init>(GeodeticCalculator.java:277)
    ...

This is sometimes, but not always, accompanied(preceded by) by:

org.jboss.resteasy.spi.UnhandledException: java.lang.IncompatibleClassChangeError: Implementing class
    ...

Examining the war files, the contents of working and broken artifacts appear to be identical, with one notable exception, the "directory ordering" of the jar files in WEB-INF/lib is different.

Performing the following procedure on the exploded war file and restarting Tomcat seems to eliminate the exception:

$ # jars in "bad" order
$ ls -U WEB-INF/lib
x.jar
b.jar
y.jar
a.jar
c.jar
z.jar
$ cp -p WEB-INF/lib/* /tmp/lib/
$ rm -r WEB-INF/lib
$ mv /tmp/lib WEB-INF/lib
$ # jars in "good" order (appears to be alphabetical after a 'cp' on my system)
$ ls -U WEB-INF/lib
a.jar
b.jar
c.jar
x.jar
y.jar
z.jar

The "good" wars don't have the jars in alphabetical order, but there appear to be a number of "good" orders a number of "bad" orders.

I initially thought we might have multiple versions of the DefaultEllipsoid class in different jars, causing a race condition between the correct version and another version, but this does not seem to be the case.

I enabled verbose classloader debugging in tomcat, and in both cases, logs/catalina.out shows this class being loaded from the correct jar:

[Loaded org.geotools.referencing.datum.DefaultEllipsoid from file: /opt/tomcat/temp/1-webapp/WEB-INF/lib/gt-referencing-11.4.jar]

Any idea of what might be going on here?

Details:

  • CentOS 7
  • Apache Tomcat/8.0.43
  • Java 1.8.0_144
  • Apache Maven 3.3.9
Chris Finley
  • 3,901
  • 5
  • 24
  • 32
  • Seems link classloader issue - I read somewhere gt jar has issues with few tomcat versions too..worth checking. – Sheetal Mohan Sharma May 11 '18 at 15:08
  • 1
    Because you can have more of one version of a class on your jars, that's why classpath order exists, to solve this issues. – LMC May 11 '18 at 15:24
  • @LuisMuñoz I'm not sure I follow, would you mind elaborating? If I understand correctly, Tomcat does not allow direct specification of the classpath order, it just has sequence of locations it searches: JVM, WEB-INF/classes, WEB-INF/lib, etc. (https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html). As far as I can tell, only one jar from my `WEB-INF/lib` directory contains the `DefaultEllipsoid` class. My problem does, however, seem to be related to the order of the files in `WEB-INF/lib` on the underlying filesystem. – Chris Finley May 15 '18 at 11:46
  • 1
    First, the class could be loaded at startup but another version could be loaded at runtime, when an instance of the class is created. Try this command on your WEB-INF/lib folder `for j in *.jar; do echo "----> $j"; jar -tvf $j | grep 'your\.class\.name' ; done` to verify the class is not present on 2 or more jars. – LMC May 15 '18 at 14:18
  • I ran your command and verified that the class only appears in a single jar. – Chris Finley May 15 '18 at 14:42
  • I saw this before, guy I worked with did not know how to build a maven project. He kept changing the versions and kept forgetting to clean his project.. This is exactly the reason why you should always use same "neutral" environment to build, check out everything straight from repo on a shared virtual box, build, deploy. – Palcente May 22 '18 at 13:23

4 Answers4

1

The line:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.geotools.referencing.datum.DefaultEllipsoid

means that the class DefaultEllipsoid is found but to be valid, there is some other class that need to be loaded but this fail. Another class is invalid.

It is this class that perhaps is duplicated with two very differents version, or one version is used for compilation, another version with different method signature at runtime.

Also, from tomcat8, the applicative jar files in WEB-INF/lib are NOT loaded according to alphabetical order anymore. I think there was a document with this on tomcat site, but for now I don't find it anymore but I found a regression bug (won't fix) on tomcat bugzilla bug 57129

This classloader stuff means that if you change some stuff on the WEB-INF/lib and restart Tomcat, then there is a bit of random class loading that make your application load one way or the other if there is duplicate jar version.

To sum up: Check the DefaultEllipsoid import and check if there is duplicate on those class. Your build also need to be cleaned to use the same version as runtime (I hope you use tools like maven to do the build)

wargre
  • 4,575
  • 1
  • 19
  • 35
  • This answer points to the key piece of information I was missing: `DefaultEllipsoid` was not the duplicated class, the duplicate class was one of its _imports_. (Actually, it was the superclass of one of its imports: `DefaultEllipsoid` imports `Ellipsoid` which extends `IdentifiedObject`, which was contained in multiple jars.) I'll bet there is a maven plugin (maven dependency plugin maybe?) that helps to detect this kind of problem if one knows how to use it properly. – Chris Finley May 22 '18 at 14:55
0

As requested by K. Sopheak and amod (moderators), I update.

There is one reason for this exception, related to Tomcat temporary directory.

Temporary files from a previous deployment could cause this. Check the workDir (default value : $TOMCAT_BASE/work), and when Tomcat is stopped clean it by removing everything.

Eugène Adell
  • 3,089
  • 2
  • 18
  • 34
  • @K.Sopheak My post is an answer to the question. I quoted a part of the question itself and as you can see, brought one answer to it. By the way, I don't understand the downvote to my answer. – Eugène Adell May 16 '18 at 14:58
  • This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stackoverflow.com/questions/ask). You can also [add a bounty](https://stackoverflow.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/low-quality-posts/19746868) – amod May 16 '18 at 15:20
  • @amod You should read the question and the answer before commenting, because obviously my answer is an answer to the question. – Eugène Adell May 16 '18 at 15:25
  • I'm still getting the same exception starting Tomcat with a completely empty workDir directory. – Chris Finley May 16 '18 at 15:54
  • @ChrisFinley what jars do you have in **$TOMCAT_BASE/lib** directory ? – Eugène Adell May 16 '18 at 16:41
  • Just the ones from the clean tomcat install: annotations-api.jar catalina-ant.jar catalina-ha.jar catalina.jar catalina-storeconfig.jar catalina-tribes.jar ecj-4.5.1.jar el-api.jar jasper-el.jar jasper.jar jaspic-api.jar jsp-api.jar servlet-api.jar tomcat-api.jar tomcat-coyote.jar tomcat-dbcp.jar tomcat-i18n-es.jar tomcat-i18n-fr.jar tomcat-i18n-ja.jar tomcat-jdbc.jar tomcat-jni.jar tomcat-util-scan.jar tomcat-util.jar tomcat-websocket.jar websocket-api.jar – Chris Finley May 16 '18 at 17:39
0

The order of the jars would really matter. basically the class loader loads them that way. So when there are multiple versions of the same class you may get this NoClassdefFoundError or java.lang.IncompatibleClassChangeError: It is able to see you classes but confused as to which one to use or the classes are modified and placed in multiple jars. Check your jars if there are any duplicate classes then you can eliminate the root cause.

vivekananda
  • 7
  • 4
  • As I mentioned in the question, and verified in the comments, the class it is complaining about is **not** contained in multiple jars. I have not verified that there are _no_ duplicate classes (in fact, I'm confident there at least a few hiding out...), but the `NoClasDefFoundError` class is only in _one_ jar. – Chris Finley May 17 '18 at 11:38
  • @vivekananda Not always, Tomcat changed its loading order. See my answer and comments. – Eugène Adell May 17 '18 at 12:42
0

Several interesting answers were given on a similar question What causes IncompatibleClassChangeError suggesting that you and the other developers are not using the very same jar files at the compilation time. Compare (at least the file size, but checksum would be better) each jar file used for the compilation, and again the ones used during the runtime, of course the ones you're using yourself and the ones used by the other developers.

Eugène Adell
  • 3,089
  • 2
  • 18
  • 34
  • Before posting the question, I did verify that the contents of `WEB-INF/lib` were identical (by comparing md5sums of the jar files) between both versions of the war, other than the directory order of the files. – Chris Finley May 17 '18 at 11:47
  • It's a very large/complicated maven project with many dependencies, so locating and comparing all of the jar files used for compilation might be a challenge, but I think you might be onto something with a difference between compile-time/run-time jars. Perhaps some kind of transitive dependency version difference or something. – Chris Finley May 17 '18 at 11:51
  • Many teams use a common repository to ensure a predictable result, particularly when the project is large/complicated. Although you clearly said you don't have twice the same class you got answers saying you have twice a same class. I don't see any other solution than checking everybody is using the same at the compilation, and then, at the packaging times. Besides, the time taken to answer a bounty question. – Eugène Adell May 17 '18 at 12:14
  • A very interesting Question and an [answer](https://stackoverflow.com/a/26642798/7748072) regarding Tomcat's jar loading order, which explains why your cp/rm/mv will change the behaviour. This suggests that a class appears twice, and one of them was compiled with a faulty library. – Eugène Adell May 17 '18 at 12:41
  • Yes - and the bug report mentioned on that answer (https://bz.apache.org/bugzilla/show_bug.cgi?id=57129) indicates I'm not the only one that Tomcat removing the sort in version 8 breaks. We do all use the same nexus repository and maven settings, so I'm reasonably confident that the versions are the same between developers. – Chris Finley May 17 '18 at 12:48
  • Could it be that the compilation is not always done with fresh .class files ? With ANT, sometimes, I faced similar problems. – Eugène Adell May 17 '18 at 12:53
  • Possible, although we almost always build with a `mvn clean install`, which will delete the target directories containing the compiled classes before it starts. There could be something stale in a local ~/.m2/repository, but I believe we've both tried clearing that out as well. Interestingly, the bad war is only produced when the source code is built on a mounted network file system, moving the code directory to the local disk and building from there produces a working artifact. – Chris Finley May 17 '18 at 19:00
  • I was thinking about something like this, since your ls -U command returns different results showing there must be different filesystem types. – Eugène Adell May 17 '18 at 19:56